Page MenuHomeFreeBSD

tcp: Rack ack war with a mis-behaving firewall or nat with resets.
ClosedPublic

Authored by rrs on Nov 11 2021, 1:02 PM.

Details

Summary

Previously we added ack-war prevention for misbehaving firewalls. This is
where the f/w or nat messes up its sequence numbers and causes an ack-war.
There is yet another type of ack war that we have found in the wild that is
like unto this. Basically the f/w or nat gets a ack (keep-alive probe or such)
and instead of turning the ack/seq around and adding a TH_RST it does something
real stupid and sends a new packet with seq=0. This of course triggers the challenge
ack in the reset processing which then sends in a challenge ack (if the seq=0 is within
the range of possible sequence numbers allowed by the challenge) and then we rinse-repeat.

This will add the needed tweaks (similar to the last ack-war prevention using the same sysctls and counters)
to prevent it and allow say 5 per second by default.

Test Plan

This can be easily tested with a pkt-drill script designed to illicit
the behavior. I will write that an insert it here soon.

Here is a packet drill script using the freebsd stack that demonstrates the issue (thank's to Michael for providing this)
rrs@lgml-rrs Downloads % cat demo_rst_ack_war.pkt
Copyright (c) 2021 Michael Tuexen
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

--ip_version=ipv4
--tolerance_usecs=250000

0.00 kldload -n tcp_bbr tcp_rack
+0.00 sysctl -w net.inet.tcp.hostcache.purgenow=1
+0.00 sysctl -w net.inet.tcp.syncookies_only=0
+0.00 sysctl -w net.inet.tcp.syncookies=1
+0.00 sysctl -w net.inet.tcp.rfc1323=1
+0.00 sysctl -w net.inet.tcp.sack.enable=1
+0.00 sysctl -w net.inet.tcp.ecn.enable=2
+0.00 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0.00 fcntl(3, F_GETFL) = 0x02 (flags O_RDWR)
+0.00 fcntl(3, F_SETFL, O_RDWR | O_NONBLOCK) = 0
+0.00 setsockopt(3, IPPROTO_TCP, TCP_FUNCTION_BLK, {function_set_name="freebsd", pcbcnt=0}, 36) = 0
+0.00 setsockopt(3, IPPROTO_TCP, TCP_KEEPIDLE, [5], 4) = 0
+0.00 connect(3, ..., ...) = -1 EINPROGRESS (Operation now in progress)
+0.00 > S 0:0(0) win 65535 <mss 1460,nop,wscale 8,sackOK,TS val 100 ecr 0>
+0.10 < S. 0:0(0) ack 1 win 32767 <mss 1440,nop,wscale 0,sackOK,eol,eol>
+0.00 > . 1:1(0) ack 1 win 15627
+0.00 send(3, ..., 1000, 0) = 1000
+0.00 > P. 1:1001(1000) ack 1 win 15627
+0.10 < . 1:1(0) ack 1001 win 32000
A keep-alive goes out
+5.00 > . 1000:1000(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+1.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
Terminate things by injecting a correct RST ACK
+0.00 < R. 1:1(0) ack 1001 win 0
+0.00 getsockopt(3, SOL_SOCKET, SO_ERROR, [ECONNRESET], [4]) = 0
+0.00 close(3) = 0

And here is a script that shows the fix limits the ack-war to 6 every second

rrs@lgml-rrs Downloads % cat fixed_rst_ack_war.pkt
Copyright (c) 2021 Michael Tuexen
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

--ip_version=ipv4
--tolerance_usecs=250000

0.00 kldload -n tcp_bbr tcp_rack
+0.00 sysctl -w net.inet.tcp.hostcache.purgenow=1
+0.00 sysctl -w net.inet.tcp.syncookies_only=0
+0.00 sysctl -w net.inet.tcp.syncookies=1
+0.00 sysctl -w net.inet.tcp.rfc1323=1
+0.00 sysctl -w net.inet.tcp.sack.enable=1
+0.00 sysctl -w net.inet.tcp.ecn.enable=2
+0.00 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0.00 fcntl(3, F_GETFL) = 0x02 (flags O_RDWR)
+0.00 fcntl(3, F_SETFL, O_RDWR | O_NONBLOCK) = 0
+0.00 setsockopt(3, IPPROTO_TCP, TCP_FUNCTION_BLK, {function_set_name="rack", pcbcnt=0}, 36) = 0
+0.00 setsockopt(3, IPPROTO_TCP, TCP_KEEPIDLE, [5], 4) = 0
+0.00 connect(3, ..., ...) = -1 EINPROGRESS (Operation now in progress)
+0.00 > S 0:0(0) win 65535 <mss 1460,nop,wscale 8,sackOK,TS val 100 ecr 0>
+0.10 < S. 0:0(0) ack 1 win 32767 <mss 1440,nop,wscale 0,sackOK,eol,eol>
+0.00 > . 1:1(0) ack 1 win 15627
+0.00 send(3, ..., 1000, 0) = 1000
+0.00 > P. 1:1001(1000) ack 1 win 15627
+0.10 < . 1:1(0) ack 1001 win 32000

A keep-alive goes out

+5.00 > . 1000:1000(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+1.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0

Keep alive now arrives

+4.00 > . 1000:1000(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0
+0.00 > . 1001:1001(0) ack 1 win 15626
A RST comes in, a challenge ACK goes out
+0.00 < R. 5:5(0) ack 1001 win 0

Keep alive now arrives

+5.00 > . 1000:1000(0) ack 1 win 15626
Terminate things by injecting a correct RST ACK
+0.00 < R. 1:1(0) ack 1001 win 0
+0.00 getsockopt(3, SOL_SOCKET, SO_ERROR, [ECONNRESET], [4]) = 0
+0.00 close(3) = 0

Diff Detail

Repository
R10 FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

rrs requested review of this revision.Nov 11 2021, 1:02 PM
rrs added a reviewer: tuexen.
sys/netinet/tcp_stacks/rack.c
14584

Why are you adding this condition?

sys/netinet/tcp_stacks/rack.c
14584

So consider.

  1. You have sent a keep-alive (forced_ack = 1)
  2. In comes a TH_ACK|TH_RST (with our seq=0) in window but we will send a challenge ack.

Now if you do go in and call rack_handle_probe_response() you will set things up so
that the RTT calculation is valid (this *may* be true or not for RTT) but you really
do not want to do that since that means our RST will clear the rtt_shift and so
the KEEP-ALIVES will go on forever counting the RST/ACK as the response
to your probe (which it may be).

We would then send 5 acks, each getting a RST/ACK which we ignore until
the next keep-alive timeout. But if you cleared all the flags the keep-alive will
be considered answered and this connection will never end.

Instead what you want to do is *not* count it as an answer so the keep-alive
will eventually clear the connection, since our limit will not do that .. it will
just keep the ack war from happening and the connection would stay forever

This revision is now accepted and ready to land.Nov 16 2021, 11:41 PM