After assigning an IP address as the local side of a point-to-point
address, the local side of the address cannot be assigned to another
interface.
Therefore, do not allow it in reverse order.
PR: 296631
Details
Details
Diff Detail
Diff Detail
- Repository
- rG FreeBSD src repository
- Lint
Lint Skipped - Unit
Tests Skipped - Build Status
Buildable 74855 Build 71738: arc lint + arc unit
Event Timeline
Comment Actions
i would rather not go with this approach as it will break my existing, functional network configuration:
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll lo1 - 90.155.69.65/32 90.155.69.65 66289 - - 0 - - ng0 - 90.155.69.65/32 90.155.69.65 1427173 - - 1459675 - -
we don't otherwise restrict assigning the same /32 to two interfaces; is there a reason it can't be allowed here?
Comment Actions
Hi, because IMHO, it's wrong.
With loopback addresses, there's functionally no difference between P2P and normal interfaces. (considering logic in ia_need_loopback_route())
Assigning the same prefix (e.g. in your case /32) to two different interfaces in the same FIB shouldn't be allowed.
We could add a sysctl to allow it, but I don't think that's necessary.
What's your actual use case here?
Comment Actions
why not?
What's your actual use case here?
the same reason you put any address on a loopback interface: so it's available when the physical interface is down (or even not present, in case of virtual interfaces like netgraph). otherwise, as soon as the interface goes down, the address is removed from the IGP and you can't connect to the router.
in FreeBSD specifically, this is also required for unnumbered IPv4 interfaces, otherwise the router will use 0.0.0.0 as the source address for ICMP errors. to fix that, the loopback /32 has to be assigned to every interface that routes IPv4 traffic. other platforms (Linux, IOS, ...) don't require this, but we do.
Comment Actions
I can confirm that having the same local PtP (Point-to-Point, in contrast to P2P peer-2-peer) address has been a working feature since the 90s, when people were still operating modem banks.
I am pretty sure I used to have two outgoing connections for most of the 2000s with the same feature (and no need for it to be on loopback but also didn't care about the [icmp] error cases in that setup).
I mean think about a big BRAS system which terminates a few thousand or tenth of thousands of ppp connections. Requiring a different local address for each connection would double-1 the amount IPv4 addresses needed (and no one would care). All the peers on the other end of the PtP link routes to your unique single IPv4 address, whereas the BRAS has an individual route to an interface for the unique remote addresses.
There are people running such setups on FreeBSD terminating L2TP/PPP sessions. They would find themselves very unhappy after an upgrade with this.
| sys/netinet/in.c | ||
|---|---|---|
| 669 | I keep wondering if what is wrong is freeing the reference; do we otherwhere make sure that if you have 3429 PtP interfaces with the same local address that the loopback route does not go away with the first one? Maybe that is what @ivy needs it for these days? Because people broke things? (I could be entirely wrong here as legacy IP is a bit of a distant memory for most for me, unless it is translation technology). | |
Comment Actions
For IGP, the best practice is to route your prefix to null/blackhole and advertise that instead.
in FreeBSD specifically, this is also required for unnumbered IPv4 interfaces, otherwise the router will use 0.0.0.0 as the source address for ICMP errors. to fix that, the loopback /32 has to be assigned to every interface that routes IPv4 traffic. other platforms (Linux, IOS, ...) don't require this, but we do.
Can you explain this please? The local address will be added as a loopback route anyway, with or without this patch.
Therefore, this should not break your use case.
Understandable, that's why I believe we can add a sysctl tunable here instead, we also talked about it in transport meeting and that's our only option for fixing those kind of bugs that exist for a long time.
Of course, I'm not convinced that we should do it here.
I am pretty sure I used to have two outgoing connections for most of the 2000s with the same feature (and no need for it to be on loopback but also didn't care about the [icmp] error cases in that setup).
I mean think about a big BRAS system which terminates a few thousand or tenth of thousands of ppp connections. Requiring a different local address for each connection would double-1 the amount IPv4 addresses needed (and no one would care). All the peers on the other end of the PtP link routes to your unique single IPv4 address, whereas the BRAS has an individual route to an interface for the unique remote addresses.
There are people running such setups on FreeBSD terminating L2TP/PPP sessions. They would find themselves very unhappy after an upgrade with this.
I'm confused how this patch might affect those BRASes. (I had those in my last ISP job for ADSL services. But I'm no longer working there.)
The loopback route is still here.
We only going to disallow setting same /32 address on multiple interfaces.
On BRAS, each user gets it's own tun and using same loopback address on multiple p2p ifps (tun) is still working since the in_addprefix() -> in_hasrtprefix() will take care of that.
For e.g:
tun0: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=4080000<LINKSTATE,MEXTPG> inet 192.168.0.1 --> 192.168.0.2 netmask 0xffffff00 groups: tun nd6 options=829<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL,STABLEADDR> tun1: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=4080000<LINKSTATE,MEXTPG> inet 192.168.0.1 --> 192.168.0.3 netmask 0xffffff00 groups: tun nd6 options=829<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL,STABLEADDR> [root@ftsr1] [~] # netstat -rn4 Routing tables Internet: Destination Gateway Flags Netif Expire default 172.23.1.254 UGS vtnet0 127.0.0.1 link#3 UH lo0 192.168.0.1 link#3 UHS lo0 192.168.0.2 link#6 UH tun0 192.168.0.3 link#7 UH tun1
| sys/netinet/in.c | ||
|---|---|---|
| 669 | Yeah, it will. that's another problem, I also mentioned it in transport meeting too, and It's hard to fix since it might affect A LOT setups out there. | |
Comment Actions
To be clear, This configuration example does NOT have local address on loopback interface. (but it has the loopback route to the lo ifp which is enough):
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Comment Actions
Because in IPv4, per RFC 3927, the only link-local scope prefix that allows this is APIPA.
Comment Actions
you have to have an IPv4 address assigned to every interface that sends IPv4 traffic. for a router, this means any interface that routes IPv4 traffic, because of ICMP error generation. an IPv4 address on the outbound interface is also required for locally-originated outbound connections that aren't bind(2)ed to a specific IP address (this applies to hosts as well as routers).
if you don't do this, the outgoing packets will have a source address of 0.0.0.0. having an IPv4 address on the loopback is not enough, it also needs to be assigned to the outgoing interface or the source selection algorithm won't use it.
in "traditional" IPv4 setups, this isn't an issue because you always have at least one link net assigned to every interface that participates in IPv4. however, modern networks are more likely to route IPv4 using IPv6 nexthops, in which case there's no need for an IPv4 address on the interface because the nexthop resolution uses IPv6 ND. but you still need to assign an address to the interface for the source selection algorithm to be happy, and in nearly all cases, you will want to use the loopback address for that.
that means a practical FreeBSD router configuration in a modern network will look something like this:
ifconfig_lo0_alias0="1.2.3.4/32" ifconfig_ix0="inet 1.2.3.4/32" ifconfig_ix0_ipv6="inet6 auto_linklocal" ifconfig_ix1="inet 1.2.3.4/32" ifconfig_ix1_ipv6="inet6 auto_linklocal" ifconfig_ix2="inet 1.2.3.4/32" ifconfig_ix2_ipv6="inet6 auto_linklocal" [ ... etc ... ]
we cannot break this unless we add another way to have functional outbound IPv4 packets on an interface without an address, e.g., something equivalent to int te0/1; ip unnumbered lo 0 in IOS, or the 'pref source' route attribute in Linux.
the original case i reported in the PR is another instance of this, the only difference is that the unnumbered IPv4 interface is a PPPoE tunnel, so it has a PtP address. but the local side of the tunnel still needs to be the loopback address - there's no other address that could be used for this. this works today as long as you assign the address to lo0 before assigning it to the tunnel.
Comment Actions
as an aside, for the ICMP case i believe you can work around this using the sysctl net.inet.icmp.reply_src=lo0 -- although i haven't tested this on a fully unnumbered interface, i would assume it works. but this doesn't affect outgoing non-ICMP connections, and it doesn't let you choose the source address per-interface, so it's not a replacement for the existing functionality.
Comment Actions
With this patch, you're still able to do this.
I've also tried your scenario to understand the concern about setting /32 twice, (but I believe it should not be related to this patch, since your example is not P2P).
here is what I got:
# netstat -rn4; ifconfig vtnet0; ifconfig lo0 Routing tables Internet: Destination Gateway Flags Netif Expire 127.0.0.1 link#3 UH lo0 172.23.1.0/24 fe80::5a9c:fcff:fe10:b89%vtnet0 UGS vtnet0 172.23.1.101 link#3 UH lo0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=880028<VLAN_MTU,JUMBO_MTU,LINKSTATE,HWSTATS> ether 58:9c:fc:09:85:57 inet 172.23.1.101 netmask 0xffffffff broadcast 172.23.1.101 inet6 fe80::5a9c:fcff:fe09:8557%vtnet0 prefixlen 64 scopeid 0x1 inet6 fdb5:c59b:114e::a prefixlen 64 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=823<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,STABLEADDR> lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> # ping -c3 172.23.1.1 PING 172.23.1.1 (172.23.1.1): 56 data bytes --- 172.23.1.1 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss
on other side:
% mdo tcpdump -evni bridge0 icmp tcpdump: listening on bridge0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 12:50:06.054363 58:9c:fc:09:85:57 > 58:9c:fc:10:0b:89, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 62845, offset 0, flags [none], proto ICMP (1), length 84) 0.0.0.0 > 172.23.1.1: ICMP echo request, id 6410, seq 274, length 64
Tried again with the address on both loopback and vtnet:
[root@ftsr1] [~] # ifconfig lo0 inet 172.23.1.101/32 [root@ftsr1] [~] # netstat -rn4; ifconfig vtnet0; ifconfig lo0; ping -c1 172.23.1.1 Routing tables Internet: Destination Gateway Flags Netif Expire 172.23.1.0/24 fe80::5a9c:fcff:fe10:b89%vtnet0 UGS vtnet0 172.23.1.101 link#3 UH lo0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=880028<VLAN_MTU,JUMBO_MTU,LINKSTATE,HWSTATS> ether 58:9c:fc:09:85:57 inet 172.23.1.101 netmask 0xffffffff broadcast 172.23.1.101 inet6 fe80::5a9c:fcff:fe09:8557%vtnet0 prefixlen 64 scopeid 0x1 inet6 fdb5:c59b:114e::a prefixlen 64 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=823<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,STABLEADDR> lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 172.23.1.101 netmask 0xffffffff inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> PING 172.23.1.1 (172.23.1.1): 56 data bytes --- 172.23.1.1 ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss
Same result.
we cannot break this unless we add another way to have functional outbound IPv4 packets on an interface without an address, e.g., something equivalent to int te0/1; ip unnumbered lo 0 in IOS, or the 'pref source' route attribute in Linux.
I will work on this feature. Thank you for letting me know.
the original case i reported in the PR is another instance of this, the only difference is that the unnumbered IPv4 interface is a PPPoE tunnel, so it has a PtP address. but the local side of the tunnel still needs to be the loopback address - there's no other address that could be used for this. this works today as long as you assign the address to lo0 before assigning it to the tunnel.
Right, PPPoE is indeed P2P. I'll setup a lab with PPPoE server + client to simulate your scenario and see what I can do about it.
I'd appreciate if you could test this patch in your lab and see if it does really affect it or not.
Comment Actions
okay, but you said:
We only going to disallow setting same /32 address on multiple interfaces.
so it seems like you intend to remove the ability to do this. have i misunderstood?
Right, PPPoE is indeed P2P. I'll setup a lab with PPPoE server + client to simulate your scenario and see what I can do about it.
I'd appreciate if you could test this patch in your lab and see if it does really affect it or not.
i can't afford a separate lab, so i only have production network to test this on. but, i will try to test the patch with a minimal configuration in a VM and see what happens.
Comment Actions
you have to delete and re-add the route after assigning the IP address.
test, on sending system:
# route add -inet 10.0.0.0/24 -inet6 2a00:1098:6b:401::11 add net 10.0.0.0: gateway 2a00:1098:6b:401::11 # ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 56 data bytes ^C --- 10.0.0.1 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss # route delete -inet 10.0.0.0/24 delete net 10.0.0.0 # ifconfig vtnet0 inet 10.1.1.1/32 # route add -inet 10.0.0.0/24 -inet6 2a00:1098:6b:401::11 add net 10.0.0.0: gateway 2a00:1098:6b:401::11 # ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 56 data bytes ^C --- 10.0.0.1 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss
on the receiving system:
10:41:06.585284 IP 0.0.0.0 > 10.0.0.1: ICMP echo request, id 46612, seq 0, length 64 10:41:07.607987 IP 0.0.0.0 > 10.0.0.1: ICMP echo request, id 46612, seq 1, length 64 10:41:08.681232 IP 0.0.0.0 > 10.0.0.1: ICMP echo request, id 46612, seq 2, length 64 10:41:34.051912 IP 10.1.1.1 > 10.0.0.1: ICMP echo request, id 49428, seq 0, length 64 10:41:35.123951 IP 10.1.1.1 > 10.0.0.1: ICMP echo request, id 49428, seq 1, length 64 10:41:36.142686 IP 10.1.1.1 > 10.0.0.1: ICMP echo request, id 49428, seq 2, length 64
the first 3 packets are without the interface address, the second three packets are with the interface address.
Comment Actions
I got confused and tested on before patch VM,, you're right.
I've the intention, but looks like we can't do it without pref-src :D
Original Fix:
[root@ftsr1] [~] # ifconfig lo0 192.168.0.1/32 [root@ftsr1] [~] # ifconfig tun0 create [root@ftsr1] [~] # ifconfig tun0 inet 192.168.0.1/32 192.168.0.2 ifconfig: ioctl (SIOCAIFADDR): File exists [root@ftsr1] [~] # ifconfig tun0 destroy [root@ftsr1] [~] # ifconfig tun0 create [root@ftsr1] [~] # ifconfig tun0 inet 192.168.0.1/32 192.168.0.2 ifconfig: ioctl (SIOCAIFADDR): File exists [root@ftsr1] [~] # ifconfig tun0 destroy [root@ftsr1] [~] # ifconfig lo0 127.0.0.1/8 [root@ftsr1] [~] # ifconfig tun0 create inet 192.168.0.1/32 192.168.0.2 [root@ftsr1] [~] # ifconfig lo0 192.168.0.1/32 ifconfig: ioctl (SIOCAIFADDR): File exists
Your case:
[root@ftsr1] [~] # ifconfig tun create inet 192.168.0.1/32 192.168.0.2 tun0 [root@ftsr1] [~] # ifconfig tun create inet 192.168.0.1/32 192.168.0.3 ifconfig: ioctl (SIOCAIFADDR): no such interface tun1
It clearly breaks it...
Right, PPPoE is indeed P2P. I'll setup a lab with PPPoE server + client to simulate your scenario and see what I can do about it.
I'd appreciate if you could test this patch in your lab and see if it does really affect it or not.i can't afford a separate lab, so i only have production network to test this on. but, i will try to test the patch with a minimal configuration in a VM and see what happens.
Thank you
Oh, so my original thought was correct, we don't need the ip address on the loopback, we only need its route to the loopback:
# netstat -rn4; ifconfig vtnet0 inet; ifconfig lo0 Routing tables Internet: Destination Gateway Flags Netif Expire 127.0.0.1 link#3 UH lo0 172.23.1.0/24 fe80::5a9c:fcff:fe10:b89%vtnet0 UGS vtnet0 172.23.1.101 link#3 UH lo0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=880028<VLAN_MTU,JUMBO_MTU,LINKSTATE,HWSTATS> inet 172.23.1.101 netmask 0xffffffff broadcast 172.23.1.101 lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> [root@ftsr1] [~] # ping -c1 -W1 172.23.1.1 PING 172.23.1.1 (172.23.1.1): 56 data bytes 64 bytes from 172.23.1.1: icmp_seq=0 ttl=64 time=0.427 ms --- 172.23.1.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.427/0.427/0.427/0.000 ms
Comment Actions
well, you have here an IPv4 address on vtnet0, so i expect this would work and source address is not 0.0.0.0. the case that doesn't work is when there is no IPv4 address on the outgoing interface.
but yes, you are correct that the IP address doesn't have to be configured on the loopback. i prefer to do this, because:
- the address is always available
- lo0 can be configured as a stub interface in OSPF to distribute a type 1 LSA for the address
even if i remove the IP address from the loopback, this doesn't solve the problem, because i need to assign the /32 to other interfaces such as Ethernet interface, wg(4) VPN interface, etc.
Comment Actions
Ok, I understand it now, let's focus on the real problem and I'll try to implement pref-src to fix it all.
Then we can discuss about PR296631.
Comment Actions
i applied your patch on my development VM and tested using tun(4) in lieu of PPPoE, and confirmed this does not allow my existing configuration:
[13!] tansy ~# ifconfig epair create epair0a [14!] tansy ~# ifconfig tun create tun0 [15!] tansy ~# ifconfig epair0a inet 10.0.0.1/32 [16!] tansy ~# ifconfig tun0 inet 10.0.0.1/32 10.0.0.2 ifconfig: ioctl (SIOCAIFADDR): File exists
(unsurprisingly, since this is the intent of the patch, so i just verified that the patch does what it says.)
i cannot allocate a different IPv4 address on the PPPoE interface because IPv4 addresses are expensive and in short supply, so we only have 64 of them and can't waste them working around implementation deficiencies.
Comment Actions
Thank you again.
i cannot allocate a different IPv4 address on the PPPoE interface because IPv4 addresses are expensive and in short supply, so we only have 64 of them and can't waste them working around implementation deficiencies.
Honestly, we should just force IPv4 deprecation already :D Would solve a lot of problems.
Comment Actions
(cross-posted)
no worries, i do happen to dislike the "assign a /32 to every interface" requirement and wish we had a better way to do this. so, if we get something like pref-src instead, i'd be very happy to start using that instead. although, i do still need to assign the /32 to the PPPoE interface somehow since a ptp address requires a local address.
incidentally, a while ago i filed a PR about this behaviour: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285422 -- there was not much engagement, though.