arp: Reduce lifetime of ARP entries

[des]

Summary

For decades, the lifetime of ARP entries has been 20 minutes. This is
known to cause issues in environments where IP adresses may get recycled
at a high rate (virtualization / cloud, wifi networks...) Reducing the
lifetime is a nop except in the rare case where we regularly exchange
packets with a given node on the local network more frequently than the
old lifetime but less frequently than the new lifetime.

• Reduce the default value of V_arpt_keep / net.link.ether.inet.max_age from 1200 seconds to 60 seconds.
• Add a V_arpt_jitter / net.link.ether.inet.jitter variable which defaults to 10 seconds.
• Combine the two to produce a random value ranging from V_arpt_keep - V_arpt_jitter to V_arpt_keep + V_arpt_jitter.

PR: 291100
MFC after: 1 week
Relnotes: yes

[des]

fix comments

[glebius]

In the past we used to have a logic that would renew ARP entires before they expire. The ARP request is sent before an entry is about to be expired. The goal is to avoid to have a doubled RTT on normal traffic every time we renew the entry. I did a quick look around, and failed to find the logic. Is it still there?

My concern is that added jitter may conflict with that. Entries might be deleted earlier than they were tried to be renewed.

[des]

In D53899#1231618, @glebius wrote:

In the past we used to have a logic that would renew ARP entires before they expire. The ARP request is sent before an entry is about to be expired. The goal is to avoid to have a doubled RTT on normal traffic every time we renew the entry. I did a quick look around, and failed to find the logic. Is it still there?

My concern is that added jitter may conflict with that. Entries might be deleted earlier than they were tried to be renewed.

I don't see how the jitter would interfere with that. The way this code has always worked is that when an entry is created or refreshed, its intended lifetime is added to the current value of time_uptime, and the result is stored in the entry as its expiry time. Any renewal logic would have to be based on that expiry time, which is still where it always was.

EDIT: btw the renewal logic is right there at the bottom of my patch, it's what wtime is for. We schedule arptimer() to be called with the entry as argument shortly before it expires.

[olce]

sys/netinet/if_ether.c

1278–1282

It seems prudent to ensure that keep is greater than 0, which is not the case when V_arpt_jitter is greater then twice V_arpt_keep (or V_arpt_keep itself is negative enough).

arc4random_uniform() expects a uint32_t, and may return a huge number if V_arpt_jitter is negative.

(Note on pre-existing code: It seems that all these variables should be of type u_int, not int.)

[glebius]

AFAIU, when arptimer() finds entry in state ARP_LLINFO_REACHABLE it gives it arp_rexmit time (1 second) to get feedback from the forwarding path and sets its state to ARP_LLINFO_VERIFY. In 1 second if feedback was collected, it will issue arprequest(). My concern is that the jitter is bigger than 1 second. I could be wrong, and my concern is erroneous. I just wanted to make sure that this specific behavior is checked to persist. I don't think we have a regression test for it.

[des]

fix jitter calculation

[des]

In D53899#1231632, @glebius wrote:

AFAIU, when arptimer() finds entry in state ARP_LLINFO_REACHABLE it gives it arp_rexmit time (1 second) to get feedback from the forwarding path and sets its state to ARP_LLINFO_VERIFY. In 1 second if feedback was collected, it will issue arprequest(). My concern is that the jitter is bigger than 1 second. I could be wrong, and my concern is erroneous. I just wanted to make sure that this specific behavior is checked to persist. I don't think we have a regression test for it.

The jitter doesn't matter. It is applied to the lifetime before we set the entry's expiry time and the renewal logic is based on the expiry time with the jitter already applied.

[bz]

In D53899#1231618, @glebius wrote:

In the past we used to have a logic that would renew ARP entires before they expire. The ARP request is sent before an entry is about to be expired. The goal is to avoid to have a doubled RTT on normal traffic every time we renew the entry. I did a quick look around, and failed to find the logic. Is it still there?

My concern is that added jitter may conflict with that. Entries might be deleted earlier than they were tried to be renewed.

I believe andre nuked those (or at least started) as it was upcalls into the TCP hostcache as well and that didn't work so well back then anymore.

To my memory, unless it got fixed, we no longer update the arp or IPv6 ND cache with availability information (at least once in a while; every packet would certainly kill a lot of performance).

So yes, I believe like you that this will increase ARP requests 20-fold and it's not only about RTT but also about extra ARP packets on a LAN.

I do believe that Linux has a mechanism to renew them in time so they can lower the cache time without thinking much.

I really wonder if the other half of the problem lies elsewhere in that we no longer send GARP when we configure IPv4 addresses?

[des]

sys/netinet/if_ether.c

1278–1282

It is already possible to set V_arpt_keep to a negative value. The result will simply be that the entry will already have expired on creation. You get what you ask for.

[bz]

In D53899#1231632, @glebius wrote:

AFAIU, when arptimer() finds entry in state ARP_LLINFO_REACHABLE it gives it arp_rexmit time (1 second) to get feedback from the forwarding path and sets its state to ARP_LLINFO_VERIFY. In 1 second if feedback was collected, it will issue arprequest(). My concern is that the jitter is bigger than 1 second. I could be wrong, and my concern is erroneous. I just wanted to make sure that this specific behavior is checked to persist. I don't think we have a regression test for it.

But the LLE logic is only updating the output path, right?

You need something on live traffic from the input path to update it when a matching packet arrives to know that the remote is still alive without sending a probe; otherwise you only have a self-perpetuating machine, right?

[olce]

sys/netinet/if_ether.c

1278–1282

Yeah, but the point is that you shouldn't be able to ask for something that silly. And a negative jitter may cause a huge jitter to be used (wrap around for arc4random_uniform() argument). The real problem here is the type of these sysctl knobs. This change doesn't make things worse, that's fine.

1278–1282

You can just call arc4random_uniform() unconditionally, it will do the right thing for an argument of 1 (and return 0 also on 0).

[cy]

Quoting the SUMMARY:

For decades, the lifetime of ARP entries has been 20 minutes. This is
known to cause issues in environments where IP adresses may get recycled
at a high rate (virtualization / cloud, wifi networks...) Reducing the
lifetime is a nop except in the rare case where we regularly exchange
packets with a given node on the local network more frequently than the
old lifetime but less frequently than the new lifetime.

Reducing the lifetime is not a nop. When the ARP entry is removed there must now be an ARP who has request must be sent. This does change the protocol in the broadest sense.

Can there be a sysctl to restore the legacy algorithm for sites that do not have many, if any, ephemeral IP addresses? Linux allows one to set this in /proc/sys/net/ipv4/neigh/ethX/gc_stale_time. We should be able to do the same in a sysctl.

[des]

sys/netinet/if_ether.c

1278–1282

Calling arc4random_uniform(1) will return 0 or 1. The right thing when V_arpt_jitter is 0 it to add 0.

1278–1282

You can _already_ ask for something that is silly.

[des]

In D53899#1231683, @cy wrote:

Can there be a sysctl to restore the legacy algorithm for sites that do not have many, if any, ephemeral IP addresses? Linux allows one to set this in /proc/sys/net/ipv4/neigh/ethX/gc_stale_time. We should be able to do the same in a sysctl.

There already is, if only you'd bothered to read the diff.

EDIT: I just realized you're referring to something else entirely, in which case can you please discuss it elsewhere?

[des]

sys/netinet/if_ether.c

1278–1282

ah sorry it will actually always return 0, right?

[cy]

In D53899#1231752, @des wrote:

In D53899#1231683, @cy wrote:

Can there be a sysctl to restore the legacy algorithm for sites that do not have many, if any, ephemeral IP addresses? Linux allows one to set this in /proc/sys/net/ipv4/neigh/ethX/gc_stale_time. We should be able to do the same in a sysctl.

There already is, if only you'd bothered to read the diff.

EDIT: I just realized you're referring to something else entirely, in which case can you please discuss it elsewhere?

I was discussing something else. Discuss it elsewhere? In a new revision maybe.

[olce]

sys/netinet/if_ether.c

1278–1282

ah sorry it will actually always return 0, right?

That's my point, yes.

[des]

simplify