Let me write a packetdrill-test to make sure we catch an KASSERT-issue.

D60444 solves the problem in a better way.

In D60416#1386523, @gallatin wrote:

I need to think about this more. For your workload, I see the benefit. For mine (busy CDN server), where we're running TCP, and where our TX rings are constantly > 80% full, I think this just adds overhead (due to full rings) and risk (due to TCP).

TCP is also interesting, because one RX packet can generate megabytes of TX. I'm not sure its appropriate at all for TCP. I'm going to add some folks from the transport side.

In D60281#1386309, @phk wrote:

I have no objection to removing the sysctl EAGAIN loop, if anything else uses it, that needs to be fixed properly.

Improve comment.

In D60281#1386054, @glebius wrote:

So what happens here is that the KPI of TCP stack switching is being changed under pressure of one of its consumers that has a special handling of EAGAIN. And we are substituting this EAGAIN to factually incorrect EINVAL. No, the argument is valid, the connection exists it is just its state that doesn't allow the switch (supposedly temporarily).

As suggested by Gleb: lie about error in sysctl_setsockopt() to avoid a loop in userland_sysctl().

Let tfb_tcp_handoff_ok() return bool. That is simpler and out of tree stacks a caught by the compiler.

An alternative approach would be to change int (*tfb_tcp_handoff_ok)(struct tcpcb *) to bool (*tfb_tcp_handoff_ok)(struct tcpcb *) and let the caller map false to EINVAL. Would that be clearer?

In D59708#1370145, @markj wrote:

In D59708#1370131, @tuexen wrote:

I think the reason for per VNET zones might have been to protect against the case where a service running in one VNET cannot run the services running in other VNETs out of resources. However, that would require per VNET limits and would need to require that this isolation is done for all resources, which doesn't seem to be the case.

Right, that is one legitimate use for per-VNET zones. In the long term, UMA itself should perhaps provide this functionality, rather than forcing consumers to create separate zones.

I think the reason for per VNET zones might have been to protect against the case where a service running in one VNET cannot run the services running in other VNETs out of resources. However, that would require per VNET limits and would need to require that this isolation is done for all resources, which doesn't seem to be the case.

In D59450#1363646, @glebius wrote:

In D59450#1363624, @tuexen wrote:

I like the idea of increasing the MTU. Is there a reason why you select 0xffff? This is the maximum size of an IPv4 packet, but an IPv6 packet can have a size of 0xffff + 40. Why not use that value? Would it make sense to use a multiple of 4?

I didn't measure, but speculatively I decided that it would be better to have it a multiple of PAGE_SIZE rather than multiple + tiny. In the second case a bulk transfer would allocate an extra page (or an mbuf, if optimized) that would be underutilized.

I like the idea of increasing the MTU. Is there a reason why you select 0xffff? This is the maximum size of an IPv4 packet, but an IPv6 packet can have a size of 0xffff + 40. Why not use that value? Would it make sense to use a multiple of 4?

In D59305#1360809, @cc wrote:

In D59305#1360805, @tuexen wrote:

What about the RACK and BBR stack?

They don't seem to be impacted or relevant.

static void rack_post_recovery(struct tcpcb *tp, uint32_t th_ack);

static void bbr_post_recovery(struct tcpcb *tp);

What about the RACK and BBR stack?

Committed in c6ae11c1b0d9.