In D18760#400670, @shurd wrote:

Looks good, only a tiny nitpick on the fail label.

In D18759#402279, @kristof wrote:

Also, I’d really, really like some basic tests for altq. It should be reasonably straightforward to do something based on the existing pf tests. At least, if altq can work on top of epairs.

That’s not a requirement to commit this patch, of course, but it’d be nice to be confident we don’t accidentally break altq.

In D18761#400698, @shurd wrote:

My only concern here is making IFLIB_MAX_RX_SEGS available to the driver...

IFLIB_MAX_RX_SEGS is more as an upper limit on what a sane driver would want rather than an upper limit of what a driver can use. If the driver actually wants (and can/will use) more, this may be the time to move ifr_frags to the end of struct iflib_rxq and use a dynamic size so the driver can have as many as it actually wants instead... that would allow other drivers to use a smaller rxq struct. ixgbe for example appears to be limited to 8 via an MPASS(), and it looks like no e1000 devices support larger than 16k frame sizes either. Dropping this to 8 for most drivers and having it at 32 or 64 for ones that support hardware 64k frames or LRO/GRO then setting this driver to use whatever is best for it could be worth the effort.

To clarify 'the symptom is that a kernel with CBQ support and not CODEL fails to load a QoS policy with the obscure error "pfctl: DIOCADDALTQ: Cannot allocate memory."', to experience this failure it is also required that the queue configuration include the BORROW flag ('borrow' keyword in pf.conf), as that sets the bit that is misinterpreted by rmclass as requesting CODEL.

Looks good to me.

In D16852#358874, @jmallett wrote:

This seems straightforward, reasonable, and sufficient. Longer-term, I'd wonder about some sort of arithmetic approach here rather than this kind of hand-scaling?

Incorporated all reviewer comments up to this point.

In D16782#357098, @kristof wrote:

It'd be really awesome if you could also see about writing some basic tests (both as pfctl parser tests, to check we can now set higher bitrates for shaping, and as a basic test that shaping actually happens), but that's certainly not blocking.

Going to commit this soon as the review has been open for nearly one month and has been idle for two weeks.

Addressed reviewer comments.

Going to commit this soon as there has not been any further review comments in nearly three weeks.

Looks good. Thanks.

Breaking into separate reviews.

In D13718#287479, @hiren wrote:

In D13718#287464, @pkelsey wrote:

In D13718#287006, @hiren wrote:

Awesome! Thank you for this work!

Now, sorry to be a PITA but you should separate out this into 3 reviews: 1) fast-open ON by default removing TCP_RFC7413 2) Bug fix for the issue in tcp_fastopen_check_cookie() 3) Client side TFO implementation.

I can do this I suppose, although I have to say it's hard for me to see the value in doing it as (2) and (3) seem pretty easy to evaluate in the current patchset, as (2) consists of moving a lock acquisition earlier in a small function and ensuring release on all the exit paths, and (3) is the mechanical removal of #ifdefs and the related rename of a kernel option that is used in one place. Perhaps because I don't think I fully understand the practical motivation for this separation, I'm also not sure whether or not it matters how I synthesize the new versions of the code - should they all be independent diffs against -current (which seems weird as it can hide interactions between the changes from review), or is there some preferred sequence (should the locking bug be presented as happening before or after all the other changes to same function)?

Motivation: easier on reviewers.
Also, I don't think you'd want to commit this as is, do you? These are 3 separate changes and I prefer 3 different commits. But as you are the one doing actual work, you can decide how to go about it. :-)
I'd get 2) and 3) committed first as they are small and then generate 1) for larger review. But again, your call. :-)

In D13718#287476, @pkelsey wrote:

In D13718#287013, @tuexen wrote:

I agree with hiren... Splitting this up in three patch sets makes a lot of sense and makes the review much simpler. While doing that, you could also change the name of the sysctl variables net.inet.tcp.fastopen.server_enabled to net.inet.tcp.fastopen.server_enable and net.inet.tcp.fastopen.client_enabled to net.inet.tcp.fastopen.client_enable to be consistent with other sysctl variables controlling the usage of features.

With respect to 'enable' vs 'enabled', I considered this issue, but I didn't find the consistency you are claiming. I saw that both are used, and I was unable to discern any rule as to when one was employed versus the other, so I picked the one that reads as a status indicator as that seemed like it would be the more frequent reason they would be read. As seen on an r327445 kernel (with my TFO patches applied, hence the grep -v), 'enabled' is used in 8 names (5 of them are loader tunables, and an overlapping 5 are runtime writeable):

$ sysctl -aN | grep -v fastopen | grep enabled | wc -l

8

In D13718#287013, @tuexen wrote:

I agree with hiren... Splitting this up in three patch sets makes a lot of sense and makes the review much simpler. While doing that, you could also change the name of the sysctl variables net.inet.tcp.fastopen.server_enabled to net.inet.tcp.fastopen.server_enable and net.inet.tcp.fastopen.client_enabled to net.inet.tcp.fastopen.client_enable to be consistent with other sysctl variables controlling the usage of features.

In D13718#287006, @hiren wrote:

Awesome! Thank you for this work!

Now, sorry to be a PITA but you should separate out this into 3 reviews: 1) fast-open ON by default removing TCP_RFC7413 2) Bug fix for the issue in tcp_fastopen_check_cookie() 3) Client side TFO implementation.

break -> continue in the addition to sbin/routed/table.c

In D10330#213998, @ae wrote:

Looks good to me. Also I want to note, that all

do 
 read();
} while()

loops are affected to the problem described in rS303374. It would be nice to fix this problem too. :)

In D9606#198724, @delphij wrote:

Previously the code is hosted at Google code and that become read-only.

I think Luigi have another repository here: https://github.com/luigirizzo/netmap-libpcap but the code haven't been updated for some time.

The main motivation is that the pcap internal interfaces changes quite frequently, and it have become more and more burdensome to maintain the compatibility shims with older pcap versions (see rS313695, for instance, that we technically need additional #ifdef's to maintain compatibility with older pcap versions).

This makes sense to me from the standpoint of the removed code being unnecessary given that it is being maintained as part of a version of libpcap that doesn't require it.

In D9465#198425, @rstone wrote:

We are trying to keep the same size and layout for a given machine type. The issue is that the TCPPCAP stuff used up uint64_t padding, and consumed different amounts of it on LP64 and LP32 systems, hence the #ifdef.

I don't have any TFO-specific complaints with removing those #ifdefs. I put them there out of uncertainty about any existing policy on userland-ABI concerns and they made it through code review. In hindsight, the #else parts of the #ifdefs should have had two items - t_Xopaque[] and t_Xspare[], because I was never of the mind that the space used by TFO might ever be used instead by something else that would then could only be enabled mutually-exclusively with TFO.

In D5017#196604, @julian wrote:

I think that "collision" is ok.

Of course it IS a new behaviour, but that is what we are trying to get. new behaviour.