Going to commit this soon as the review has been open for nearly one month and has been idle for two weeks.

Addressed reviewer comments.

Going to commit this soon as there has not been any further review comments in nearly three weeks.

Looks good. Thanks.

Breaking into separate reviews.

In D13718#287479, @hiren wrote:

In D13718#287464, @pkelsey wrote:

In D13718#287006, @hiren wrote:

Awesome! Thank you for this work!

Now, sorry to be a PITA but you should separate out this into 3 reviews: 1) fast-open ON by default removing TCP_RFC7413 2) Bug fix for the issue in tcp_fastopen_check_cookie() 3) Client side TFO implementation.

I can do this I suppose, although I have to say it's hard for me to see the value in doing it as (2) and (3) seem pretty easy to evaluate in the current patchset, as (2) consists of moving a lock acquisition earlier in a small function and ensuring release on all the exit paths, and (3) is the mechanical removal of #ifdefs and the related rename of a kernel option that is used in one place. Perhaps because I don't think I fully understand the practical motivation for this separation, I'm also not sure whether or not it matters how I synthesize the new versions of the code - should they all be independent diffs against -current (which seems weird as it can hide interactions between the changes from review), or is there some preferred sequence (should the locking bug be presented as happening before or after all the other changes to same function)?

Motivation: easier on reviewers.
Also, I don't think you'd want to commit this as is, do you? These are 3 separate changes and I prefer 3 different commits. But as you are the one doing actual work, you can decide how to go about it. :-)
I'd get 2) and 3) committed first as they are small and then generate 1) for larger review. But again, your call. :-)

In D13718#287476, @pkelsey wrote:

In D13718#287013, @tuexen wrote:

I agree with hiren... Splitting this up in three patch sets makes a lot of sense and makes the review much simpler. While doing that, you could also change the name of the sysctl variables net.inet.tcp.fastopen.server_enabled to net.inet.tcp.fastopen.server_enable and net.inet.tcp.fastopen.client_enabled to net.inet.tcp.fastopen.client_enable to be consistent with other sysctl variables controlling the usage of features.

With respect to 'enable' vs 'enabled', I considered this issue, but I didn't find the consistency you are claiming. I saw that both are used, and I was unable to discern any rule as to when one was employed versus the other, so I picked the one that reads as a status indicator as that seemed like it would be the more frequent reason they would be read. As seen on an r327445 kernel (with my TFO patches applied, hence the grep -v), 'enabled' is used in 8 names (5 of them are loader tunables, and an overlapping 5 are runtime writeable):

$ sysctl -aN | grep -v fastopen | grep enabled | wc -l

8

In D13718#287013, @tuexen wrote:

I agree with hiren... Splitting this up in three patch sets makes a lot of sense and makes the review much simpler. While doing that, you could also change the name of the sysctl variables net.inet.tcp.fastopen.server_enabled to net.inet.tcp.fastopen.server_enable and net.inet.tcp.fastopen.client_enabled to net.inet.tcp.fastopen.client_enable to be consistent with other sysctl variables controlling the usage of features.

In D13718#287006, @hiren wrote:

Awesome! Thank you for this work!

Now, sorry to be a PITA but you should separate out this into 3 reviews: 1) fast-open ON by default removing TCP_RFC7413 2) Bug fix for the issue in tcp_fastopen_check_cookie() 3) Client side TFO implementation.

break -> continue in the addition to sbin/routed/table.c

In D10330#213998, @ae wrote:

Looks good to me. Also I want to note, that all

do 
 read();
} while()

loops are affected to the problem described in rS303374. It would be nice to fix this problem too. :)

In D9606#198724, @delphij wrote:

Previously the code is hosted at Google code and that become read-only.

I think Luigi have another repository here: https://github.com/luigirizzo/netmap-libpcap but the code haven't been updated for some time.

The main motivation is that the pcap internal interfaces changes quite frequently, and it have become more and more burdensome to maintain the compatibility shims with older pcap versions (see rS313695, for instance, that we technically need additional #ifdef's to maintain compatibility with older pcap versions).

This makes sense to me from the standpoint of the removed code being unnecessary given that it is being maintained as part of a version of libpcap that doesn't require it.

In D9465#198425, @rstone wrote:

We are trying to keep the same size and layout for a given machine type. The issue is that the TCPPCAP stuff used up uint64_t padding, and consumed different amounts of it on LP64 and LP32 systems, hence the #ifdef.

I don't have any TFO-specific complaints with removing those #ifdefs. I put them there out of uncertainty about any existing policy on userland-ABI concerns and they made it through code review. In hindsight, the #else parts of the #ifdefs should have had two items - t_Xopaque[] and t_Xspare[], because I was never of the mind that the space used by TFO might ever be used instead by something else that would then could only be enabled mutually-exclusively with TFO.

In D5017#196604, @julian wrote:

I think that "collision" is ok.

Of course it IS a new behaviour, but that is what we are trying to get. new behaviour.

This is the source comment that should have been submitted with my 'request for change' action, had I pressed the buttons in a way that didn't dispose of it instead.

Adjusted label names and comments

In D8218#170900, @jtl wrote:

In D8218#170887, @pkelsey wrote:

This is functionally equivalent, but I don't think it is clearer. I think it is less clear that the conditional decrement is placed after the tfo_done label as it will never execute after a goto tfo_done.

I placed it there as a matter of future-proofing. It doesn't hurt anything for it to be after the tfo_done label. But, if someone makes a code change to add a new goto tfo_done, it will still hit this check.

Because you obviously have strong feelings for exactly how this code should be (and this doesn't really impact me), I've opened a bug and assigned it to you. You can fix it at your convenience.

This is functionally equivalent, but I don't think it is clearer. I think it is less clear that the conditional decrement is placed after the tfo_done label as it will never execute after a goto tfo_done. There is clear one-to-one correspondence between creating a TFO socket and not needing this decrement that I think is blurred by this positioning because the decrement is not an action that is common to 'done' and 'tfo_done', it is unique to 'done'. I think we would be better off if the code wasn't arranging to cancel a conditional action because it was placed in a path it never needs to be in.

In D8218#170623, @jtl wrote:

In D8218#170606, @pkelsey wrote:

I can only identify one circumstance where the t_tfo_pending counter on a listen socket will be incremented without a corresponding decrement ever occurring - when all of the following conditions are met:

1. TFO is enabled in the system
2. A TFO SYN with an invalid TFO cookie matches a listen socket that has TFO enabled
3. The current t_tfo_pending count on that socket is <= so_qlimit / 2

The other way is...

1. TFO enabled
2. t_tfo_pending count <= so_qlimit / 2
3. SYN without a TFO option, followed by a SYN with a valid TFO cookie and the same IPs and port numbers

In D8218#170606, @pkelsey wrote:

See line-specific comments in the patch.

I don't see in-line comments. Perhaps you can resubmit them?

There is still a path that leaks when mac_syncache_init() fails.

I can only identify two circumstances where the t_tfo_pending counter on a listen socket will be incremented without a corresponding decrement ever occurring.

In D7701#159878, @jhb wrote:

This looks correct though I'd be surprised if any code actually ran into it. (Or at least, if kthread_add() is failing your system is probably in trouble already.)

Absent any issues raised, I am going to commit this sometime in the next 24-48 hours.

Fixed issue with discarding TFO ACKs in tcp_do_segment() and fixed issue with snd_una not being properly advanced when TFO connections transition to ESTABLISHED, also in tcp_do_segment(). Also some minor cleanups (comments, whitespace, etc).

File contents for tcp_fastopen.[ch] were missing from the initial diff.

I don't think we should be taking any

We shouldn't be taking any action at all based on a NEEDFRAG message that suggests (or from which we deduce) a new mtu that is the same or larger than the current one. I think the clearest expression of this idea is to only call tcp_mtudisc() from tcp_ctlinput() when the new mtu could decrease tp->t_maxseg (and remove the comment that tcp_mtudisc() 'does the right thing').