netinet, netinet6: wait for the net epoch before freeing the lltable

[olivier]

Summary

in_ifdetach() and in6_ifdeparture() freed the lltable while the ifnet was
still reachable by index, so an RTF_LLDATA RTM_DELETE (or netlink
RTM_DELNEIGH) racing with interface destruction used freed memory.
Clear the pointer, NET_EPOCH_WAIT(), then free, as before 0d469d23715d.

Fixes: 0d469d23715d ("net: attach IPv4 and IPv6 stacks to an interface with EVENTHANDLER(9)")

Test Plan

Was detected while running an experimental version of net/frr10 full regression test (topotest).
I’ve reproduced it with inet only, and while checking the inet6 code I detected the same problem (but didn’t try to reproduce).

KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame 0xfffffe0115ade700
 vpanic() at vpanic+0x149/frame 0xfffffe0115ade830
 panic() at panic+0x43/frame 0xfffffe0115ade890
 trap_fatal() at trap_fatal+0x6f/frame 0xfffffe0115ade8b0
 calltrap() at calltrap+0x8/frame 0xfffffe0115ade8b0
 --- trap 0x9, rip = 0xffffffff80c2a189, rsp = 0xfffffe0115ade980, rbp = 0xfffffe0115ade9c0 ---
 __mtx_lock_flags() at __mtx_lock_flags+0x49/frame 0xfffffe0115ade9c0
 lltable_delete_addr() at lltable_delete_addr+0x2e/frame 0xfffffe0115ade9f0
 rts_send() at rts_send+0x8d1/frame 0xfffffe0115adebd0
 sosend_generic_locked() at sosend_generic_locked+0x5ab/frame 0xfffffe0115adec80
 sosend_generic() at sosend_generic+0x61/frame 0xfffffe0115adecd0
 sousrsend() at sousrsend+0x79/frame 0xfffffe0115aded30
 dofilewrite() at dofilewrite+0x82/frame 0xfffffe0115aded80
 sys_write() at sys_write+0x132/frame 0xfffffe0115adedf0
 amd64_syscall() at amd64_syscall+0x17c/frame 0xfffffe0115adef30

[glebius]

The NET_EPOCH_WAIT() is actually a crutch. I would agree with this change if we were very close to stable/16. For main, I'd like to make it right. The table shall be freed with deferred free. Do you have a good repro for this panic? I can create this change.

[glebius]

Was detected while running an experimental version of net/frr10 full regression test (topotest).

I guess I need to install the package. What next?

[olivier]

In D60787#1394240, @glebius wrote:

Was detected while running an experimental version of net/frr10 full regression test (topotest).

I guess I need to install the package. What next?

It will be not engouh, because need some custom frr patches, but I will prepare a "one script" while looking for a simpler solution not requiering frr.
The frr test was doing:
Test steps:

1. test_bgp_route_install_r1: sharpd on r1 installs 2000 routes (sharp install routes 39.99.0.0 nexthop ... 2000), which BGP advertises to r2.
2. test_bgp_established: all 128 BGP sessions come up.
3. test_bgp_routes_on_r2: r2 holds the 2000 routes with 128-way ECMP, and all of them share one nexthop group (NHG) of 128 members. Checking that NHG is what the test is about.
4. test_bgp_shutdown_some_links: set r2-eth30 to r2-eth49 down. It checks that 20 peers fail and the NHG shrinks. This step fails on FreeBSD ("Route updates skipped did not increase"), but that failure isn't the kernel bug. The interfaces are only set down here, not destroyed, so no crash yet.
5. Teardown, where it crashes:
   • stopRouter() should stop the daemons first, but its liveness check uses /proc, which FreeBSD doesn't mount. It concludes every daemon is already dead, so zebra and bgpd keep running.
   • munet then destroys all the epairs and bridges in parallel (ifconfig ... destroy).
   • For each interface that disappears, the kernel sends zebra a departure message. Zebra handles it with if_nbr_ipv6ll_to_ipv4ll_neigh_del_all(), which deletes the RFC 5549 neighbor 169.254.0.1 on that interface.
   • With the reporter's patch, that delete becomes a routing-socket RTM_DELETE with RTF_LLDATA.
   • zebra is fast, so its delete arrives while that interface is still being detached: in_ifdetach() has already freed the IPv4 data and ARP table, but the interface number still resolves (ifnet_byindex()). The kernel locks the freed table and panics.
     
     Why it happens every time
     
     Each teardown destroys 128 interfaces while zebra answers each departure within microseconds, so each run gives about 128 chances to hit a window that is only a few microseconds wide. The reporter's (a frr dev) dump (30+ ifconfig in ifnet_de) and mine (13 ifconfig, zebra running) were both taken in the middle of that parallel destroy.

[olivier]

In D60787#1394240, @glebius wrote:

Was detected while running an experimental version of net/frr10 full regression test (topotest).

I guess I need to install the package. What next?

script here:
https://github.com/ocochard/myscripts/blob/master/FreeBSD/frr-topotest-vm/repro-in_ifdetach-uaf.sh

[olivier]

In D60787#1394240, @glebius wrote:

Was detected while running an experimental version of net/frr10 full regression test (topotest).

I guess I need to install the package. What next?

here is a simpler reproducer:
https://github.com/ocochard/myscripts/blob/master/FreeBSD/frr-topotest-vm/lltable-detach-uaf.c