Why are you removing the data after close checks in BBR and Rack. This is not related to getting a FIN (where you may
still be open). The point of that is so that if we have closed after dumping data into the SB, and then the peer sends in
a request (not realizing we have closed yet) to NOT send a RST until after all the data is through. This was useful for
NF since often times Nginx will close() a connection after filling the N requests and finishing dumping data into the sb.
It say has 1Meg in the buffer when it does close().

Add Drew's read_frequently compiler directive.

Update to Gleb and I's agreed upon reduction at the if in system.h

In D44420#1015031, @glebius wrote:

My bad, didn't understood proposed change. Of course if pointer stays stable since module load, there is no race.

In D44420#1014931, @glebius wrote:

In D44420#1014587, @rrs wrote:

Why not just use

if (hpts_that_need_softclock > 0)

tcp_hpts_softclock()\

?

But that would create the same race you mentioned above, wouldn't it? IMHO, using function pointer variable is the way to go and it potentially has a chance to get reduced to fewer instructions if compiler gets smarter.

Another issue we will need to address is multiple hpts threads trying to set the tcp_hpts_softclock pointer. One can envision
a case where you have one that is just lowering it to zero, and so goes into the "set" to NULL block. And another thread that
is going from 0 -> 1 and so goes into the set to function block.

Why not just use

In D44420#1013898, @glebius wrote:

Why can't we just clear and set the function pointer itself? That would reduce impact on userret() down to just one instruction.

I do believe the extern is not needed here since systm.h is included.

Adopt Drew's optimization suggestion and reverted back the sysctl Michael pointed out since the arg is unused (and here I thought
all these years it was the default).

Adopt Michael's suggestion to re-use one of the available bits for the flag.

Updated to use an atomic instead of the u64_counter as Gleb suggests.

and yeah Drew we could add a block with sysctl

I can easily change to an atomic.. I did not realize the fetch was expensive :)

Add in the additional MPASS as Gleb suggested.

Good catch Gleb, I missed the hpts.c changes.. I will update it :=)

So in testing I found a slight bug in that we were on the connect() side setting up the PCM max seg to
early. This means we end up with 10 x 512 not 10 x mss. Lets fix it so just like the listening side things get
deferred until we have gotten to the established state.

The hpts change is now in https://reviews.freebsd.org/D44157

This separates out the hpts fix. Will add that as a new review after I drop this in. I have
tested the combined hpts fix and this with the current head and everything works as
expected :)

I still need to re-test this after the last sync I did.. will do that and then split out the hpts change and update the diff before landing..

Ok update to after the sync. Also I forgot to add in the inherit calls in the syncache and the tcp_usr_attach

In D43986#1003519, @rrs wrote:

In D43986#1003451, @tuexen wrote:

Did you try git add tcp_pcm.c? Please note that this week is stabilization week, meaning only bug fixes should go in...

No I did not, but once I do that will it show up in the diff?

I will give it a try and see.

In D43986#1003451, @tuexen wrote:

Did you try git add tcp_pcm.c? Please note that this week is stabilization week, meaning only bug fixes should go in...

Fix a bug I found in testing (and I think Gleb just pointed out)

Glad to see this :)

One comment, Hans had a hand in restructuring the code a while ago too :)

After looking at this and thinking on it changing the structure is the *wrong* approach since it leaves a landmine in place
that can get someone if they don't include opt_inet.h. The better choice is to make all things that ifdef in the tcppcb to
be in opt_global.h so that the entire kernel always knows what tcpcb looks like.

Why if 0x10000 is not used would it matter?