Avoid IPv6 source address selection on accepting TCP connections

[ae]

Summary

The problem and different from IPv4 behavior was found with ipfw fwd to IPv6 address that is configured in jail.
We have some application that works in jail and listens port 13333 for both IPv4 and IPv6. On host system we forward all TCP connections to port 3333 to address and port in the jail. With IPv4 it works as expected. With IPv6 we see successful TCP handshake and then server sends RST just after final ACK.

# tcpdump -ni lagg0 tcp and port 3333
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on lagg0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:32:33.410088 IP6 fc00::2.63582 > fc00::6.3333: Flags [S], seq 275485310, win 64800, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0
09:32:33.410112 IP6 fc00::6.3333 > fc00::2.63582: Flags [S.], seq 1400678893, ack 275485311, win 65535, options [mss 1440,nop,wscale 6,sackOK,eol], length 0
09:32:33.410235 IP6 fc00::2.63582 > fc00::6.3333: Flags [.], ack 1, win 8235, length 0
09:32:33.410328 IP6 fc00::6.3333 > fc00::2.63582: Flags [R], seq 1400678894, win 0, length 0

With enabled net.inet.tcp.log_debug=1 we can see these messages in the log:

Sep  1 12:32:33 btr-test kernel: TCP: [fc00::2]:63582 to [fc00::6]:3333; syncache_socket: in6_pcbconnect failed with error 49
Sep  1 12:32:33 btr-test kernel: TCP: [fc00::2]:63582 to [fc00::6]:3333 tcpflags 0x10<ACK>; tcp_input_with_port: Listen socket: Socket allocation failed due to limits or memory shortage, sending RST

Further debugging shows that errors is returned through in6_pcbladdr() -> in6_selectsrc_socket() -> in6_selectsrc() -> prison_local_ip6().
And it is because our source IPv6 address doesn't belongs to the jail, since fwd_tag was used to find corresponding listen socket.

The patch makes behavior to be similar to IPv4. in6_pcbconnect() can be called from syncache_socket() on accepting established connection, and from UDP&TCP connect(), when we are initiate connection. We can reuse bool argument rehash to determine our case and skip IPv6 SAS.

Test Plan

[btr-test butcher]# ifconfig lo99 create
[btr-test butcher]# ifconfig lo99 inet6 fc00::dead:beef/128
[btr-test butcher]# ifconfig lo99 inet 10.9.8.66/32
[btr-test butcher]# ifconfig lagg0 inet6 fc00::6/64
[btr-test butcher]# sysctl net.inet6.ip6.forwarding=1
net.inet6.ip6.forwarding: 0 -> 1
[btr-test butcher]# kldload ipfw
[btr-test butcher]# ipfw add fwd fc00::dead:beef,13333 ip6 from any to any 3333 proto tcp
00100 fwd fc00::dead:beef,13333 ip6 from any to any 3333 proto tcp
[btr-test jail]# cat jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

path = "/home/jail-test";
host.hostname = "jail-test";

jail_test {
        jid = 22;
        interface = lo99;
        ip6.addr = fc00::dead:beef;
        ip4.addr = 10.9.8.66;
}

[btr-test jail]# jail -cf jail.conf
jail_test: created
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Clearing /tmp (X related).
Updating /var/run/os-release done.
Creating and/or trimming log files.
Updating motd:.
Starting syslogd.
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.

Thu Aug 31 14:54:27 UTC 2023
[btr-test jail]# sockstat -lj 22
USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sshd        1557 3   tcp6   fc00::dead:beef:13333 *:*
root     sshd        1557 4   tcp4   10.9.8.66:13333       *:*
root     syslogd     1505 5   udp6   fc00::dead:beef:514   *:*
root     syslogd     1505 6   udp4   10.9.8.66:514         *:*
root     syslogd     1505 7   dgram  /var/run/log <-
root     syslogd     1505 8   dgram  /var/run/logpriv

In this test sshd was used to listen on 13333 port.
On client:

% nc -6 fc00::6 3333
SSH-2.0-OpenSSH_9.4 FreeBSD-20230810
^D