Change Details

This can be easily tested with a pkt-drill script designed to illicit the behavior. I will write that an insert it here soon.

This can be easily tested with a pkt-drill script designed to illicit the behavior. I will write that an insert it here soon. Here is a packet drill script using the freebsd stack that demonstrates the issue (thank's to Michael for providing this) rrs@lgml-rrs Downloads % cat demo_rst_ack_war.pkt // Copyright (c) 2021 Michael Tuexen // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions // are met: // 1. Redistributions of source code must retain the above copyright // notice, this list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright // notice, this list of conditions and the following disclaimer in the // documentation and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE // ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE // FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL // DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS // OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) // HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT // LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY // OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF // SUCH DAMAGE. // --ip_version=ipv4 --tolerance_usecs=250000 0.00 `kldload -n tcp_bbr tcp_rack` +0.00 `sysctl -w net.inet.tcp.hostcache.purgenow=1` +0.00 `sysctl -w net.inet.tcp.syncookies_only=0` +0.00 `sysctl -w net.inet.tcp.syncookies=1` +0.00 `sysctl -w net.inet.tcp.rfc1323=1` +0.00 `sysctl -w net.inet.tcp.sack.enable=1` +0.00 `sysctl -w net.inet.tcp.ecn.enable=2` +0.00 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 +0.00 fcntl(3, F_GETFL) = 0x02 (flags O_RDWR) +0.00 fcntl(3, F_SETFL, O_RDWR | O_NONBLOCK) = 0 +0.00 setsockopt(3, IPPROTO_TCP, TCP_FUNCTION_BLK, {function_set_name="freebsd", pcbcnt=0}, 36) = 0 +0.00 setsockopt(3, IPPROTO_TCP, TCP_KEEPIDLE, [5], 4) = 0 +0.00 connect(3, ..., ...) = -1 EINPROGRESS (Operation now in progress) +0.00 > S 0:0(0) win 65535 <mss 1460,nop,wscale 8,sackOK,TS val 100 ecr 0> +0.10 < S. 0:0(0) ack 1 win 32767 <mss 1440,nop,wscale 0,sackOK,eol,eol> +0.00 > . 1:1(0) ack 1 win 15627 +0.00 send(3, ..., 1000, 0) = 1000 +0.00 > P. 1:1001(1000) ack 1 win 15627 +0.10 < . 1:1(0) ack 1001 win 32000 // A keep-alive goes out +5.00 > . 1000:1000(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +1.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // Terminate things by injecting a correct RST ACK +0.00 < R. 1:1(0) ack 1001 win 0 +0.00 getsockopt(3, SOL_SOCKET, SO_ERROR, [ECONNRESET], [4]) = 0 +0.00 close(3) = 0 And here is a script that shows the fix limits the ack-war to 6 every second rrs@lgml-rrs Downloads % cat fixed_rst_ack_war.pkt // Copyright (c) 2021 Michael Tuexen // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions // are met: // 1. Redistributions of source code must retain the above copyright // notice, this list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright // notice, this list of conditions and the following disclaimer in the // documentation and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE // ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE // FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL // DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS // OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) // HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT // LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY // OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF // SUCH DAMAGE. // --ip_version=ipv4 --tolerance_usecs=250000 0.00 `kldload -n tcp_bbr tcp_rack` +0.00 `sysctl -w net.inet.tcp.hostcache.purgenow=1` +0.00 `sysctl -w net.inet.tcp.syncookies_only=0` +0.00 `sysctl -w net.inet.tcp.syncookies=1` +0.00 `sysctl -w net.inet.tcp.rfc1323=1` +0.00 `sysctl -w net.inet.tcp.sack.enable=1` +0.00 `sysctl -w net.inet.tcp.ecn.enable=2` +0.00 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 +0.00 fcntl(3, F_GETFL) = 0x02 (flags O_RDWR) +0.00 fcntl(3, F_SETFL, O_RDWR | O_NONBLOCK) = 0 +0.00 setsockopt(3, IPPROTO_TCP, TCP_FUNCTION_BLK, {function_set_name="rack", pcbcnt=0}, 36) = 0 +0.00 setsockopt(3, IPPROTO_TCP, TCP_KEEPIDLE, [5], 4) = 0 +0.00 connect(3, ..., ...) = -1 EINPROGRESS (Operation now in progress) +0.00 > S 0:0(0) win 65535 <mss 1460,nop,wscale 8,sackOK,TS val 100 ecr 0> +0.10 < S. 0:0(0) ack 1 win 32767 <mss 1440,nop,wscale 0,sackOK,eol,eol> +0.00 > . 1:1(0) ack 1 win 15627 +0.00 send(3, ..., 1000, 0) = 1000 +0.00 > P. 1:1001(1000) ack 1 win 15627 +0.10 < . 1:1(0) ack 1001 win 32000 // // A keep-alive goes out // +5.00 > . 1000:1000(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +1.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 // // Keep alive now arrives // +4.00 > . 1000:1000(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 // // Keep alive now arrives // +5.00 > . 1000:1000(0) ack 1 win 15626 // Terminate things by injecting a correct RST ACK +0.00 < R. 1:1(0) ack 1001 win 0 +0.00 getsockopt(3, SOL_SOCKET, SO_ERROR, [ECONNRESET], [4]) = 0 +0.00 close(3) = 0

This can be easily tested with a pkt-drill script designed to illicit the behavior. I will write that an insert it here soon. Here is a packet drill script using the freebsd stack that demonstrates the issue (thank's to Michael for providing this) rrs@lgml-rrs Downloads % cat demo_rst_ack_war.pkt // Copyright (c) 2021 Michael Tuexen // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions // are met: // 1. Redistributions of source code must retain the above copyright // notice, this list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright // notice, this list of conditions and the following disclaimer in the // documentation and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE // ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE // FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL // DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS // OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) // HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT // LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY // OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF // SUCH DAMAGE. // --ip_version=ipv4 --tolerance_usecs=250000 0.00 `kldload -n tcp_bbr tcp_rack` +0.00 `sysctl -w net.inet.tcp.hostcache.purgenow=1` +0.00 `sysctl -w net.inet.tcp.syncookies_only=0` +0.00 `sysctl -w net.inet.tcp.syncookies=1` +0.00 `sysctl -w net.inet.tcp.rfc1323=1` +0.00 `sysctl -w net.inet.tcp.sack.enable=1` +0.00 `sysctl -w net.inet.tcp.ecn.enable=2` +0.00 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 +0.00 fcntl(3, F_GETFL) = 0x02 (flags O_RDWR) +0.00 fcntl(3, F_SETFL, O_RDWR | O_NONBLOCK) = 0 +0.00 setsockopt(3, IPPROTO_TCP, TCP_FUNCTION_BLK, {function_set_name="freebsd", pcbcnt=0}, 36) = 0 +0.00 setsockopt(3, IPPROTO_TCP, TCP_KEEPIDLE, [5], 4) = 0 +0.00 connect(3, ..., ...) = -1 EINPROGRESS (Operation now in progress) +0.00 > S 0:0(0) win 65535 <mss 1460,nop,wscale 8,sackOK,TS val 100 ecr 0> +0.10 < S. 0:0(0) ack 1 win 32767 <mss 1440,nop,wscale 0,sackOK,eol,eol> +0.00 > . 1:1(0) ack 1 win 15627 +0.00 send(3, ..., 1000, 0) = 1000 +0.00 > P. 1:1001(1000) ack 1 win 15627 +0.10 < . 1:1(0) ack 1001 win 32000 // A keep-alive goes out +5.00 > . 1000:1000(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +1.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // Terminate things by injecting a correct RST ACK +0.00 < R. 1:1(0) ack 1001 win 0 +0.00 getsockopt(3, SOL_SOCKET, SO_ERROR, [ECONNRESET], [4]) = 0 +0.00 close(3) = 0 And here is a script that shows the fix limits the ack-war to 6 every second rrs@lgml-rrs Downloads % cat fixed_rst_ack_war.pkt // Copyright (c) 2021 Michael Tuexen // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions // are met: // 1. Redistributions of source code must retain the above copyright // notice, this list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright // notice, this list of conditions and the following disclaimer in the // documentation and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE // ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE // FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL // DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS // OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) // HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT // LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY // OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF // SUCH DAMAGE. // --ip_version=ipv4 --tolerance_usecs=250000 0.00 `kldload -n tcp_bbr tcp_rack` +0.00 `sysctl -w net.inet.tcp.hostcache.purgenow=1` +0.00 `sysctl -w net.inet.tcp.syncookies_only=0` +0.00 `sysctl -w net.inet.tcp.syncookies=1` +0.00 `sysctl -w net.inet.tcp.rfc1323=1` +0.00 `sysctl -w net.inet.tcp.sack.enable=1` +0.00 `sysctl -w net.inet.tcp.ecn.enable=2` +0.00 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 +0.00 fcntl(3, F_GETFL) = 0x02 (flags O_RDWR) +0.00 fcntl(3, F_SETFL, O_RDWR | O_NONBLOCK) = 0 +0.00 setsockopt(3, IPPROTO_TCP, TCP_FUNCTION_BLK, {function_set_name="rack", pcbcnt=0}, 36) = 0 +0.00 setsockopt(3, IPPROTO_TCP, TCP_KEEPIDLE, [5], 4) = 0 +0.00 connect(3, ..., ...) = -1 EINPROGRESS (Operation now in progress) +0.00 > S 0:0(0) win 65535 <mss 1460,nop,wscale 8,sackOK,TS val 100 ecr 0> +0.10 < S. 0:0(0) ack 1 win 32767 <mss 1440,nop,wscale 0,sackOK,eol,eol> +0.00 > . 1:1(0) ack 1 win 15627 +0.00 send(3, ..., 1000, 0) = 1000 +0.00 > P. 1:1001(1000) ack 1 win 15627 +0.10 < . 1:1(0) ack 1001 win 32000 // // A keep-alive goes out // +5.00 > . 1000:1000(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +1.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 // // Keep alive now arrives // +4.00 > . 1000:1000(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 +0.00 > . 1001:1001(0) ack 1 win 15626 // A RST comes in, a challenge ACK goes out +0.00 < R. 5:5(0) ack 1001 win 0 // // Keep alive now arrives // +5.00 > . 1000:1000(0) ack 1 win 15626 // Terminate things by injecting a correct RST ACK +0.00 < R. 1:1(0) ack 1001 win 0 +0.00 getsockopt(3, SOL_SOCKET, SO_ERROR, [ECONNRESET], [4]) = 0 +0.00 close(3) = 0