Add GARP retransmit capability
ClosedPublic
Actions

Authored by dab on Aug 29 2016, 4:24 PM.

Details

Reviewers

badger
vangyzen

Group Reviewers

network

Commits

rS306577: Add GARP retransmit capability

Summary

A single gratuitous ARP (GARP) is always transmitted when an IPv4
address is added to an interface and that is usually
sufficient. However in some circumstances, such as when a shared
address is passed between cluster nodes, this single GARP may
occasionally be dropped or lost. This can lead to neighbors on the
network link working with a stale ARP cache and sending packets
destined for that address to the node that previously owned the
address, which may not respond.

To avoid this situation GARP retransmits can be enabled by setting
the net.link.ether.inet.garp_rexmit_count sysctl to a value greater
than zero. The setting represents the maximum number of
retransmissions. The interval between retransmissions is calculated
using an exponential backoff algorithm, doubling each time, so the
retransmission intervals are: {1, 2, 4, 8, 16, ...} (seconds).

Due to the exponential backoff algorithm used for the interval between
GARP retransmissions, the maximum number of retransmissions is limited
to 16 for sanity. This limit corresponds to a maximum interval between
retransmissions of 2^16 seconds ~= 18 hours. Increasing this limit is
possible, but sending out GARPs spaced days apart would be of little
use.

Test Plan

Ran this script:

#!/bin/sh
GARP_SYSCTL=net.link.ether.inet.garp_rexmit_count
IF=em0
ADDR=172.23.115.88
NETADDR="${ADDR}/24"
ADDR2=172.23.115.89
NETADDR2="${ADDR2}/24"

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') $*"
}

log "Starting tcpdump to display GARPs"
tcpdump -vvv -i $IF "(arp host ${ADDR}) or (arp host ${ADDR2})" &
DUMPPID=$!

log "Current value if $GARP_SYSCTL:"
sysctl $GARP_SYSCTL

log "Setting value of ${GARP_SYSCTL} to 0"
sysctl ${GARP_SYSCTL}=0
log "Adding address ${NETADDR} to $IF: expect a single GARP"
ifconfig $IF inet ${NETADDR} alias
sleep 10
log "Removing address ${NETADDR} from $IF"
ifconfig $IF inet ${NETADDR} -alias

log "Setting value of ${GARP_SYSCTL} to 4"
sysctl ${GARP_SYSCTL}=4

log "Adding address ${NETADDR} to $IF: expect 5 GARPs"
ifconfig $IF inet ${NETADDR} alias
sleep 32
log "Removing address ${NETADDR} from $IF"
ifconfig $IF inet ${NETADDR} -alias

log "Adding address ${NETADDR} to $IF: expect 3 GARPs (address will be deleted before all 5 can be sent)"
ifconfig $IF inet ${NETADDR} alias
sleep 6
log "Removing address ${NETADDR} from $IF"
ifconfig $IF inet ${NETADDR} -alias

(
    log "Adding address ${NETADDR} to $IF: expect 5 GARPs"
    ifconfig $IF inet ${NETADDR} alias
    sleep 32
    log "Removing address ${NETADDR} from $IF"
    ifconfig $IF inet ${NETADDR} -alias
) &

(
    log "Adding address ${NETADDR2} to $IF: expect 3 GARPs (address will be deleted before all 5 can be sent)"
    ifconfig $IF inet ${NETADDR2} alias
    sleep 6
    log "Removing address ${NETADDR2} from $IF"
    ifconfig $IF inet ${NETADDR2} -alias
) &

sleep 35
kill $DUMPPID

log "Interface $IF should be back to original address(es) now:"
ifconfig $IF

log "Test done"

Got this output showing proper operation:

2016-08-29 11:16:34 Starting tcpdump to display GARPs
2016-08-29 11:16:34 Current value if net.link.ether.inet.garp_rexmit_count:
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
net.link.ether.inet.garp_rexmit_count: 4
2016-08-29 11:16:34 Setting value of net.link.ether.inet.garp_rexmit_count to 0
net.link.ether.inet.garp_rexmit_count: 4 -> 0
2016-08-29 11:16:34 Adding address 172.23.115.88/24 to em0: expect a single GARP
11:16:34.893874 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
2016-08-29 11:16:44 Removing address 172.23.115.88/24 from em0
2016-08-29 11:16:44 Setting value of net.link.ether.inet.garp_rexmit_count to 4
net.link.ether.inet.garp_rexmit_count: 0 -> 4
2016-08-29 11:16:44 Adding address 172.23.115.88/24 to em0: expect 5 GARPs
11:16:45.002455 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:16:46.065847 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:16:48.089488 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:16:52.089662 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:17:00.089678 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
2016-08-29 11:17:17 Removing address 172.23.115.88/24 from em0
2016-08-29 11:17:17 Adding address 172.23.115.88/24 to em0: expect 3 GARPs (address will be deleted before all 5 can be sent)
11:17:17.100189 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:17:18.156096 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:17:20.189011 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
2016-08-29 11:17:23 Removing address 172.23.115.88/24 from em0
2016-08-29 11:17:23 Adding address 172.23.115.88/24 to em0: expect 5 GARPs
2016-08-29 11:17:23 Adding address 172.23.115.89/24 to em0: expect 3 GARPs (address will be deleted before all 5 can be sent)
11:17:23.207698 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:17:23.214390 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.89 tell 172.23.115.89, length 28
11:17:24.240741 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.89 tell 172.23.115.89, length 28
11:17:24.240825 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:17:26.251366 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:17:26.251443 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.89 tell 172.23.115.89, length 28
2016-08-29 11:17:29 Removing address 172.23.115.89/24 from em0
11:17:30.250293 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
11:17:38.289407 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.115.88 tell 172.23.115.88, length 28
2016-08-29 11:17:55 Removing address 172.23.115.88/24 from em0

17 packets captured
124 packets received by filter
0 packets dropped by kernel
2016-08-29 11:17:58 Interface em0 should be back to original address(es) now:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0c:29:a2:6b:3b
inet6 fe80::20c:29ff:fea2:6b3b%em0 prefixlen 64 scopeid 0x1
inet 172.16.211.154 netmask 0xffffff00 broadcast 172.16.211.255
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
2016-08-29 11:17:58 Test done

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 4968
Build 5034: arc lint + arc unit

Event Timeline

dab updated this revision to Diff 19795.Aug 29 2016, 4:24 PM

dab retitled this revision from to Add GARP retransmit capability..

dab updated this object.

dab edited the test plan for this revision. (Show Details)

dab added reviewers: vangyzen, badger.

Herald added a subscriber: imp. · View Herald TranscriptAug 29 2016, 4:24 PM

dab edited the test plan for this revision. (Show Details)Aug 29 2016, 4:27 PM

dab added a reviewer: network.

@badger, @vangyzen:

If you compare this with my similar change in our local highlander tree, you will find a couple primary differences:

Different locking --- the address list locking changed from 10.x (and prior) to 12.0-CURRENT
Change to conditions under which the ifa_free() call is made after callout_stop() call in in.c/in_difaddr_ioctl (). This was due to a change in the callout API from 10.x (and prior) to 12.0-CURRENT. Until I inserted the test for callout_stop returning 1 (it was a simple boolean check before), I was crashing with a "use after free" error because I would call ifa_free() too many times.

dab retitled this revision from Add GARP retransmit capability. to Add GARP retransmit capability.Aug 29 2016, 5:18 PM

badger added inline comments.Aug 30 2016, 2:43 AM

sys/netinet/if_ether.c
1321	Can't arg1 be NULL? I notice that sysctl_handle_int() begins with if (arg1)

badger added inline comments.Aug 30 2016, 2:51 AM

sys/netinet/if_ether.c
1321	Ugh, nevermind. Too late at night to be looking at this :).

Could you please explain why this should be done in kernel?

Given that you probably already have some script for adding and IP address to the new node (based on CARP or other quorum selection mechanisms) - you can easily generate these packets in userland.
For example, arp-sk (available from ports) can be used to send GARPs using simple one-liner.

I think several things point to this belonging in the kernel:

It's a fairly fundamental responsibility of the network stack to generate a GARP when an address is added. The first (and currently only) GARP is generated by the kernel.
Pushing it to user land doesn't fix the general case. Each application/system would have to implement a solution.
While the particular circumstance that gave rise to this solution was on a floating cluster address, I do not think that the potential problem exists only for such a circumstance. It seems that this could happen when adding any address under any circumstances.
The generation and retransmission of the IPv4 GARP is similar in nature to IPv6 ND/DAD, which is also done in the kernel.

vangyzen accepted this revision.Oct 1 2016, 11:52 PM

vangyzen edited edge metadata.

This revision is now accepted and ready to land.Oct 1 2016, 11:52 PM

Closed by commit rS306577: Add GARP retransmit capability (authored by vangyzen). · Explain WhyOct 2 2016, 1:43 AM

This revision was automatically updated to reflect the committed changes.