The SMR-protected inpcb lookup algorithm currently has to check whether
a matching inpcb belongs to a jail, in order to prioritize jailed
bound sockets. To do this it has to maintain a ucred reference, and for
this to be safe, the reference can't be released until the UMA
destructor is called, and this will not happen within any bounded time
period.
Changing SMR to periodically recycle garbage is not trivial. Instead,
let's try to implement SMR-synchronized lookup without needing to
dereference inp_cred.
Commit 220d89212943 ("inpcb: immediately return matching pcb on lookup")
gets us part of the way there. This patch handles lookups of
unconnected sockets. Here, the strategy is to maintain some order of
items within a hash chain so that a wild lookup can simply return the
first match and preserve existing semantics. This makes insertion of
listening sockets more complicated in order to make lookup simpler,
which seems like the right tradeoff anyway given that bind() is already
a fairly expensive operation.
In particular, when inserting an unconnected socket, in_pcbinhash() now
keeps the following ordering:
- jailed sockets before non-jailed sockets
- specified local addresses before unspecified local addresses
Most of the change adds a separate SMR-based lookup path for inpcb hash
lookups. When a match is found, we try to lock the inpcb and
re-validate its connection info. In the common case, this works well
and we can simply return the inpcb. If this fails, typically because
something is concurrently modifying the inpcb, we go to the slow path,
which performs a serialized lookup.
Note, I did not touch lbgroup lookup, since there the credential
reference is formally synchronized by net_epoch, not SMR. In
particular, lbgroups are rarely allocated or freed.
I think it is possible to simplify in_pcblookup_hash_wild_locked() now,
but I didn't do it in this patch.
Sponsored by: Klara, Inc.
Sponsored by: Modirum MDPay