Since r286722, IPv4 LLE address field is updated while holding AFDATA WLOCK.
In order to remove LLE read lock we need to do a couple of other things.
First, some sort of flags (which can be safely used w/o lle lock) needs to be added to check if we really can use given address data
Second, we need to provide some sort of feedback if the entry is really used when expiration time is approaching. If we skip doing this
we will end up either infinitely keeping all lle records or expiring them even they are used atm.
First is easily solved by adding another flag field updated under AFDATA WLOCK along with mac address.
Second one is more tricky.
Currently, when packet is sent using given lle and its expiration time is within lle->la_preempt = V_arp_maxtries = 5 seconds, we decrease la_preempt under lle RLOCK and send arprequest towards dst. If traffic flow continues (and int case of no reply) we will send up to V_arp_maxtries (one each second) before expiring entry.
New behavior:
We Introduce a simple state machine (incomplete->reachable<->verify->deleted). arptimer is scheduled V_arp_maxtries seconds ealier. On first timer call in "reachable" state code sets special r_skip_req field to 1, sets state to "verify" and reschedule itself V_arpt_rexmit = 1 second forward. While looking up lle fast path checks if given r_skip_req field is nonzero and sets it to zero under new lle req mutex (which should be very rare). On subsequent calls in arptimer we check if r_skip_req was changed (and issue arprequest if yes) and reschedule until expire (if valid reply was received, set state back to "reachable" and reschedule arptimer according to V_arpt_keep value.
Given that, introduce 2 new fields:
- r_flags, containing flags used by fast path code. New flags are: RLLE_VALID (pre-compiled yes/no flag if we can use the entry) RLLE_IFADDR (to be able to pass given flag as hint to ether_output)
- r_skip_req, used for fast path feedback.
The new 'req_mtx' mutex is introduced to guarantee very short and deterministic lock/unlock period (lle locks _can_ be holded much longer like in in_lltable_dump_entry) which is crucial to switching lltable lock to rmlock.