ipsec: replace SECASVAR mtx by rmlock

[kp]

Summary

This mutex is a significant point of contention in the ipsec code, and
can be relatively trivially replaced by a read-mostly lock.
It does require a separate lock for the replay protection, which we do
here by adding a separate mutex.

This improves throughput (without replay protection) by 10-15%.

Sponsored by: Orange Business Services

[kp]

This is a largely mechanical substitution, there's going to be room for further improvement, but it's a somewhat significant gain already so it's worth doing anyway.

There are some places where the current locking looks odd to me, but I've not yet been able to dig further. This change doesn't improve that, but also doesn't make it any worse.

[mjg]

sys/netipsec/xform_ah.c

968

this is an example why the patch in the current form does not work: replay count gets bumped without the lock by many cpus at the same time. all places of the sort will need to be coverted to use atomics. Specifically this one will unfortunately need to be a fcmpset loop to make sure you don't bump across the limit.

i was looking at replacing this lock myself and my general idea was to use sequence counters. it would avoid degeneration from IPIs on write locking (although I don't know how frequently that's needed) and be faster single-threaded at least on amd64. However, it may happen to be slower on arm64 as it requires 2 load fences.

[kp]

Ensure that all uses of 'replay' are protected by the replay mutex.