Fix race when accepting TCP connections

[tuexen]

Summary

When expanding a SYN-cache entry to a socket/inp a two step approach was taken. The local addresses were filled in, then the inp was added to the hash table. In a second step, the remote addresses were filled in and the inp was relocated in the hash table. A write lock is held when this happens and the code looking up entries was getting a corresponding read lock. Since the read lock is gone away since the introduction of the epochs, the half filled inp was found during lookup. This resulted in the following traces:

10:02:04.186021 IP 192.168.1.45.30496 > 192.168.1.203.80: Flags [S], seq 3911499413, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2011789803 ecr 0], length 0
10:02:04.186081 IP 192.168.1.203.80 > 192.168.1.45.30496: Flags [S.], seq 216370171, ack 3911499414, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 813333713 ecr 2011789803], length 0
10:02:04.188493 IP 192.168.1.45.30496 > 192.168.1.203.80: Flags [.], ack 1, win 1026, options [nop,nop,TS val 2011789806 ecr 813333713], length 0
10:02:04.188566 IP 192.168.1.45.30498 > 192.168.1.203.80: Flags [S], seq 3763588859, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1426831841 ecr 0], length 0
10:02:04.188697 IP 192.168.1.203.80 > 192.168.1.45.30498: Flags [.], ack 3911499414, win 1026, options [nop,nop,TS val 813333716 ecr 2011789806], length 0
10:02:04.188791 IP 192.168.1.45.30496 > 192.168.1.203.80: Flags [P.], seq 1:69, ack 1, win 1026, options [nop,nop,TS val 2011789806 ecr 813333713], length 68: HTTP: GET /missing HTTP/1.1
10:02:04.189035 IP 192.168.1.203.80 > 192.168.1.45.30496: Flags [P.], seq 1:304, ack 69, win 1026, options [nop,nop,TS val 813333716 ecr 2011789806], length 303: HTTP: HTTP/1.1 404 Not Found
10:02:04.191778 IP 192.168.1.45.30498 > 192.168.1.203.80: Flags [R], seq 3911499414, win 0, length 0
10:02:04.192124 IP 192.168.1.45.30496 > 192.168.1.203.80: Flags [F.], seq 69, ack 304, win 1026, options [nop,nop,TS val 2011789809 ecr 813333716], length 0
10:02:04.192178 IP 192.168.1.203.80 > 192.168.1.45.30496: Flags [F.], seq 304, ack 70, win 1026, options [nop,nop,TS val 813333719 ecr 2011789809], length 0
10:02:04.194897 IP 192.168.1.45.30496 > 192.168.1.203.80: Flags [.], ack 305, win 1026, options [nop,nop,TS val 2011789813 ecr 813333719], length 0
10:02:05.189887 IP 192.168.1.45.30498 > 192.168.1.203.80: Flags [S], seq 3763588859, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1426832842 ecr 0], length 0
10:02:05.189919 IP 192.168.1.203.80 > 192.168.1.45.30498: Flags [S.], seq 3917148958, ack 3763588860, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3330366102 ecr 1426832842], length 0
10:02:05.193038 IP 192.168.1.45.30498 > 192.168.1.203.80: Flags [.], ack 1, win 1026, options [nop,nop,TS val 1426832845 ecr 3330366102], length 0
10:02:05.193222 IP 192.168.1.45.30498 > 192.168.1.203.80: Flags [P.], seq 1:69, ack 1, win 1026, options [nop,nop,TS val 1426832845 ecr 3330366102], length 68: HTTP: GET /missing HTTP/1.1
10:02:05.193813 IP 192.168.1.203.80 > 192.168.1.45.30498: Flags [P.], seq 1:304, ack 69, win 1026, options [nop,nop,TS val 3330366106 ecr 1426832845], length 303: HTTP: HTTP/1.1 404 Not Found
10:02:05.196745 IP 192.168.1.45.30498 > 192.168.1.203.80: Flags [F.], seq 69, ack 304, win 1026, options [nop,nop,TS val 1426832849 ecr 3330366106], length 0
10:02:05.196816 IP 192.168.1.203.80 > 192.168.1.45.30498: Flags [F.], seq 304, ack 70, win 1026, options [nop,nop,TS val 3330366109 ecr 1426832849], length 0
10:02:05.199794 IP 192.168.1.45.30498 > 192.168.1.203.80: Flags [.], ack 305, win 1026, options [nop,nop,TS val 1426832852 ecr 3330366109], length 0

This patch changes the above procedure in a way that the inp is fully filled before inserted in the hash table.

Thanks to Paul (devgs at ukr dot net) for reporting this issue on the [e-mail] mailing list.

Test Plan

Run

wrk -c 10 --header "Connection: close" -d 60 -t 1 --latency "http://[fe80::20d:b9ff:fe41:cd2d%igb1]:80/missing"

and

wrk -c 10 --header "Connection: close" -d 60 -t 1 --latency  http://192.168.2.202:80/missing

(with updated IP addresses) against a web server (nginx, for example) on a host with using a NIC supporting multiple ithreads (like igb).