Looks good to me :-)

Adds Jonathan's code (with one minor modification to make it print right).

This updates the latest round from Jonathan.

The code currently in head will return one of three values:

I can be convinced *not* to have a user space program (though that is exactly what unix was founded upon, a
bunch of small programs that did things for you that could be put together.. I am an old timer so I harken
from those days)..

For some reason fabricator is not cooperating with me. sigh.

I still wonder why we need a new user-space program, when you could use sysctls to do the same thing.

• Well if you look at an email Hiren asked me .. how do I tell the number of pcb's that are there attached to it? The -l option does that for you.. the rest is just fluff... I am pretty set on having a user space function at this point, it also allows us to add more things in the future. So stop wondering and go with the flow :-)

This updates for the latest comments and adds back the missing
user program that I inadvertently took out of the diff (by not
diffing in the right place :-D)

Ok after reviewing a bit of the tcp_function_set routine, I updated it to
have a default no arguments to just show the default, the -l to
show the default with a *. No changes to the TCP stack itself here.. I think
we are about ready.. :-)

I think the SCTP change are ok, I will flag this to Michael when you commit it (the code
in the FreeBSD netinet directory is not the base code..). The key here is its *very* doubtful
that SCTP and IPSEC can work. This is due to the multi-homing and the fact that once you
turn on IPSEC you would not be able to do un-ordered or partial reliability. There is
nothing in the SCTP stack to restrict this so, if a user tried this the association would fail.
DTLS is the right answer for SCTP not IPSEC.. as exemplified in all the deployment of
SCTP over DTLS over UDP over Stun/Ice in firefox/chrome using our SCTP code ;-)

In another review, Han's commented on having source code in modules. Since
it probably is not a good idea lets move them to a sub-dir called netinet/tcp_stacks
and then just arrange the modules makefile to point over there.

Just in case, lets go ahead and move all the sources
under

I have no idea to that Hans though it works well :-)

Adds and updates comments. I did find a small difference below
the comments in the auto-buffer sizing so that too was fixed.

Update the man page per Han's suggestion.

fix my comment and I think you will be good to go

The socket buffer with SCTP is just not something thats workable. There are
all sorts of pre-defined notions that closely align a socket buffer to stream-of-bytes
semantics of TCP. With UDP its never an issue, since you have all un-ordered who
cares up come the messages.

Ok this will update it so we get three return codes out
of callout_stop and friends

This updates all things from Jonathan's comments..

Ok this addresses getting all things in a module (framework and callout code test for
new async-drain tests). And it also moves headers to sys/tests

Gesh, I am going to get more coffee.. .pick the right update silly.

opps forgot the arg's in the man

Ok this adds the small section into the manual page. With
that we are about ready :-)

all the rest I will fix shortly

This update is now tested. However I still need to an update to
the manual. Kib has a proposal to change where I KASSERT(drain == NULL) and
I think we should do that i.e. make that case return -1.

Oh I forgot to mark this :-)

Hmm the problem here is the same I see in building the async-drain functionality. This
is the one case that is rather nebulous in the callout return when stopping. If you
return 1, you say the callout was stopped.. but was it? Not really, this block of
code will happen in the case you illustrate but also in the case where the
callout was already stopped. Returning 1 when it was already stopped (that I was able
to stop it) is not really what you want.

Fix imp's blank line style issue.

Fix the great bug Han's found in setting drain..

Ok lets make it so that *always* if you call callout_async_drain() on a stopped/completed timer
then kassert and die. You need to always assure the timer is running before you call
async_drain!

After chatting with jhb, some changes he suggested.

Fixes typo Han's caught.. great catch thanks!

After thinking about it, we need to move the clear of active down and
only call drain in the duplicate stop case if the caller did not
clear active (which they should have when the callout really completed).

The right way to do this is so that
callout_stop_async_drain(struct callout *, drain_function)
is made.

Opps, I blew that by having timers defined for the default side. Lets take this
chance to re-structure so the top 3 functions are the ones that must be defined
and optional functions follow with comments.

Ok I think this wraps up the comments. We will want to re-address
the timer issue if we get a callout_drain_async() in the timer world since
without out it we end up adding a *bunch* of stuff to try to maintain the
rather twisty timer based callout_drain_async() that tcp is maintaining :-)

Ok this wraps up the timer concern, however this does not yet address
a issue we have with the timer_stop() call.. i.e. the hookey way we stop since
a callout_drain_async() is not present in the callout system. How do we stop all
and assure the flags are set properly??

I disagree with you imp

Thanks for your review.. I did not expect you to look that klm.. its just a fast path copy
of something I have been playing with. I wanted to be able to show how it worked to
give folks something to play with.

To me this looks good. You are properly computing the pipe per RFC 6675, and yes
that does mean you double count (at times) some of the data as being in-flight. The
RFC actually specifically states that :-)

Should not the callout have some sort of lock?

I don't know but as yet, I have not taken the time to think through the problem. I don't
understand why you think there is a *burning* need for this call. TCP has, as you pointed
out, used a similar method.. it can do that and *not* harm itself since it knows what it
expects the lock to return as (locked or unlocked) and take appropriate action.

The big difference here is if I set a typical sysctl in FreeBSD I can easily shoot-myself and
thats ok. I pay the price for my stupid setting of the sysctl. The big difference with some of
the TCP settings is that not only I, but those connected to my area of the Internet also will
pay the price.

So Lawrence,

This has a very very awful side effect in it. You change
the callout lock type. This means the KAPI user may have
specified to return locked, and you change it to UNLOCKED.

We don't use TOE (we use LRO though). The panic's we have are the persist timer. Lawrence
has an idea though and is investigating that. Maybe he can turn something up.

Hi all:

I do still want to look at this a bit deeper though before we land this, we probably should
add lstewart to the review too.

This is an excellent fix for a very nasty problem thats been lurking in the tcp stack for quite some time ;-)

This updates us to:
a) What Hiren is running in production i.e. split the c_flags into two, and avoid a lock in callout_deactivate()
b) Take JHB's suggestion and have non-caring macro calls to the per_cpu base calls *not* pass in c->c_cpu but

instead a cookie, in this case -1, I did not want to use CPUBLOCK since that is already over-loaded.

Hmm well thats the one that jhb objected to, to complex to look at I think was his
reasoning. It works fine .. IMO.. of course to make it "simpler" then you have to
add the lock which is a bit more overhead but I can't measure the difference on
my systems.

ok so it has the callout_lock() inside callout_deactivate().. perfect.. that is what
I will commit this weekend :-)
R

Ok I have tested the former method (separate flags) that Hiren currently is using, and I
have tested the method suggested by jhb that uses a lock (shown here). There is
no detectible performance difference that I can see. The same machine with a
similar traffic load ran 40Gbps on both nights with virtually the same CPU load metrics.
Of course our load is all elephants with almost no mice, but it probably will be
the same for large numbers of mice since you most likely hope these timers
do *not* go off and you don't call deactivate unless the timer goes off...

As of this coming Friday I will finish up my testing.. I am going to
contrast the performance penalty we get by using the lock in the deactivate
vs splitting the field.. (all my testing so far as been with a split field for the most part).

This addresses Han's issue with falling back to the "previous" CPU.. instead we
fall forward to the new one. The only way this occurs with CPUBLOCK in place
is if another thread holds the lock and is doing a cpu_switch.. so we just update
the lcpu to have the new value before we do the switch.

This update takes JHB's suggestion and instead of splitting the
c_flags into two fields, makes callout_deallocate() be a function and it
gets the CC_LOCK for any bit twiddling.

Separate the internal state keeping flags to a new c_iflags. Also fix
ng which incorrectly accesses the flags directly and was failing to use
the macros (which do the right thing).

For Han's pointed out callout_stop/callout_reset issue with migration. We can't
re-order the test, since then stop would fail in the locked case.

Ok after much discussion with Hans, we *could* have an issue where the
user sends in an invalid CPU. This is *not* what I think is happening with Hiren
since the cc_cpu and lock is all sane (it would be a invalid index to cc_cpu which
would not have an init'd lock). But I have created a new review since this one
is already committed

I have thought long and hard about this. I don't think its a bug.
But to know for sure I will need to add some instrumentation.

I think your wrong here. The caller of callout_cpu_switch() is holding
the CC_LOCK(). Now there are only two callers of this function.