Good point on sub structures.

If the cb_destroy virtual function shouldn't null the pointer, I assert that it shouldn't free it either.

The latest diff is based on feedback from @jhb via @sbruno :

fix a potential use after free in access to inp_options

use correct constants

I agree that using less macros didn't particularly help with readability. But there are in fact two problems with the current code. The issue you pointed out with regard to protecting a user space controlled kernel memory allocation, and secondly, properly detecting and recovering from the requested data length changing when the in6p is reacquired.

Fix GET_PKTOPT_VAR bugs with memory allocation.

minor update

@melifaro this is probably what you were thinking?

refactor GET_PKTOPT_VAR into a function

I believe I have addressed all concerns, primarily by more fully understanding the functions that were interrelated and realizing that there wasn't anywhere near as much to fix in the first place. The earlier revisions of this patch were far reaching, but now the fixes are just internal to tcp_ipsec_pcbctl()

@melifaro I was not able to refactor into functions but I was able to refactor into something easier to follow and more correct.

properly rebase

update GET_PKTOPT_VAR to properly check for pktopt change after dropping lock

fix scope of options to match that of len

Looks good to me.

I agree that having the default stack as a module would be useful for A/B testing the default stack and pushing out fixes without rebooting, and is sufficiently valuable for inclusion into FreeBSD.

Any radical departure from the current behavior of basically creating and copying struct tcp_info to user space would require a different getsockopt() optname at a bare minimum, and we'd still have to support the current TCP_INFO getsockopt() anyway.

malloc and ip options are a bigger problem, going back to the drawing board.

@melifaro I'm thinking about how to make GET_PKTOPT_VAR a function or perhaps... how to make it much smaller and call a function for the bulk of the logic.

additional fixes

I will start a new review to address feedback from @melifaro, this one is broken.

fix malloc length in ip6_output.c:GET_PKTOPT_VAR

fix typo

use long for len passed to malloc in ip_ctloutput() for getting IP_RETOPTS

fix len passed to malloc in ip_ctloutput() for getting IP_RETOPTS

fix locking within tcp_ipsec_pcbctl() to match ipsec4_pcbctl(), ipsec4_pcbctl()

In D14623#307351, @ae wrote:

Why you need INP_WLOCK() for getsockopt()? Also ipsec_get_pcbpolicy() requieres INP_WLOCK(), and IPv4 version uses WLOCK, but IPv6 version uses RLOCK.

fix IPv6 IPSEC_PCBCTL locking to be consistent with IPv4

I am abandoning this in favor of 6 smaller reviews, per @rwatson's general advice to break up the patch.

use flag instead of cached value for old max_cwnd tracking in cubic

In D14141#299658, @jason_eggnet.com wrote:

In D14141#299456, @hiren wrote:

In D14141#299284, @jason_eggnet.com wrote:

In D14141#299283, @hiren wrote:

@jason_eggnet.com why did you move K calculation from post recovery to ack received? I don't think that's correct.

So basically, why not delay the calculation of K until right before it is needed. Use the previous value of K if neither of the inputs have changed.

I am hoping this is just the 'debugging' code to find out why we are getting a stale value of K and not something to commit.
Or I am thoroughly confused at this point. :-)

I think you're confused :) I've updated the commit in the review formally, let me know if you have any questions, I think I've detailed it reasonably well.

off by one with overflow threshold in cubic_cwnd

In D14141#299456, @hiren wrote:

In D14141#299284, @jason_eggnet.com wrote:

In D14141#299283, @hiren wrote:

@jason_eggnet.com why did you move K calculation from post recovery to ack received? I don't think that's correct.

So basically, why not delay the calculation of K until right before it is needed. Use the previous value of K if neither of the inputs have changed.

I am hoping this is just the 'debugging' code to find out why we are getting a stale value of K and not something to commit.
Or I am thoroughly confused at this point. :-)

Fix underflow and overflow conditions in the cubic cc

In D14141#299283, @hiren wrote:

@jason_eggnet.com why did you move K calculation from post recovery to ack received? I don't think that's correct.

In D14141#298565, @lstewart wrote:

@jason_eggnet.com I thought about this some more and while there is no doubt that overflow/underflow due to the function's inputs is possible and needs to be remedied, the cause of overflow in your test case is not in fact the time since congestion being too large, but the bogus value of K, which for a wmax of 1460 bytes (i.e. slightly more than 1 MSS) should be 205 per my quick check:

In D14141#297959, @hiren wrote:

In D14141#297953, @jason_eggnet.com wrote:

In D14141#297944, @hiren wrote:

I remember this one. :-)
jegg, can you please elaborate on 'cwnd < 1smss bad for other parts of the code' part?
Thanks.

A zero cwnd can cause infinite looping (i.e., continual goto again; looping) within tcp_output(). Technically even one byte would be ok but there's no real point to having the cwnd be some fractional mss value.

In general the concept of having a zero cwnd doesn't make any sense.

oh, I know 0 won't do any good. :-)

I meant why not 0 < cwnd < mss. I know we generally stick to multiple of mss for cwnd but just wondering what goes wrong if we don't. (Specially when we are bytes based as opposed to packets based wrt cwnd).

Thanks.

In D14141#297944, @hiren wrote:

I remember this one. :-)
jegg, can you please elaborate on 'cwnd < 1smss bad for other parts of the code' part?
Thanks.

In D14141#296724, @jtl wrote:

Is this truly an underflow? Or, is it a low number caused by an overflow in one of the earlier calculations? (It might help if you could give a precise example from your testing.)

The distinction is important because this might not be the correct solution to an overflow in one of the earlier calculations.

Fix memory leak in ip_ctloutput; add documentation to pcbctl function.

Missed INP_WUNLOCK() in ipv6 setsockopt() processing

remove in_pcbref in ctl functions

It looks like a socket is guaranteed to have a single defined inp that is valid (specifically, not free'd) from the time socket or accept is called all the way to the time that close is called.

allow for inp / in6p being null in certain cases within ip_ctl functions

It looks like there might be cases in ip_ctloutput, ip6_ctloutput, rip_ctloutput, and ip6_raw_ctloutput where the inp/in6p being null is ok.

update outstanding issues

Sounds good, I'll abandon this.

No problem, thanks :)

Ok, Sean just reverted the previous patch, and this patch is based on the new HEAD.

rebase patch to new HEAD

We'll probably be abandoning this review in favor of keeping the current behavior and ensuring sendfile behaves the same as send and write.

Gleb wants us to hold off on this for the moment, with the first step of making sendfile() work like current send() / write() paths.

• Consistently handle locking in tcp_ipsec_pcbctl(), as well as use rlocks for reads, and wlocks for writes.

In D12586#261038, @ae wrote:

Probably you need to modify netipsec/ipsec_pcb.c too.

This issue was discovered by rare memory corruption when calling ip6_optlen(). The likely culprit was unlocked manipulation of in6p->in6p_outputopts.

rename OPTSET2292_REQUIRED to OPTSET2282_EXCLUSIVE

Before patch:

Using packetdrill with this https://github.com/nplab/packetdrill/pull/137

Abandoning this due to insufficient proof of a problem.