Page MenuHomeFreeBSD

Relase LLE table resources before ND6.
Needs ReviewPublic

Authored by ae on Nov 2 2021, 10:13 PM.

Details

Reviewers
melifaro
Group Reviewers
network
Summary

In the system where ifnets are often destroyed it is possible to
catch the race, when ND6 did already freed its resources, but ND6
callout timer fires nd6_llinfo_timer and it tries to get access
to ND6 data in the ifnet's if_afdata. This leads NULL pointer
dereference. Avoid this by freeing LLE table before ND6 resources.

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint OK
Unit
No Unit Test Coverage
Build Status
Buildable 42533
Build 39421: arc lint + arc unit

Event Timeline

ae requested review of this revision.Nov 2 2021, 10:13 PM

This leads to a lot of questions:
(a) can we check for the NULL pointer and gracefully handle it? Or at least add a KASSERT to document it?
(b) I guess I should go and look where the nd6_llinfo_timer() comes from,,, or let me ask: what drains the callout?

Changing the order to me sounds like hiding a different problem (which likely is a missing callout_drain)?

In D32810#740413, @bz wrote:

This leads to a lot of questions:
(a) can we check for the NULL pointer and gracefully handle it? Or at least add a KASSERT to document it?

Yes, this diff initially was part of D32811, @melifaro asked me to make it separate.
The panic that I'm trying to fix was in the line ndi = ND_IFINFO(ifp);:

Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 0c
fault virtual address	= 0x10
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d2b21c
stack pointer	        = 0x28:0xfffffe00d7e399c0
frame pointer	        = 0x28:0xfffffe00d7e39a50
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi4: clock (0))
trap number		= 12
panic: page fault
cpuid = 6
time = 1633476489
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00d7e39680
vpanic() at vpanic+0x182/frame 0xfffffe00d7e396d0
panic() at panic+0x43/frame 0xfffffe00d7e39730
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00d7e39790
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00d7e397e0
trap() at trap+0x271/frame 0xfffffe00d7e398f0
calltrap() at calltrap+0x8/frame 0xfffffe00d7e398f0
--- trap 0xc, rip = 0xffffffff80d2b21c, rsp = 0xfffffe00d7e399c0, rbp = 0xfffffe00d7e39a50 ---
nd6_llinfo_timer() at nd6_llinfo_timer+0x9c/frame 0xfffffe00d7e39a50
softclock_call_cc() at softclock_call_cc+0x139/frame 0xfffffe00d7e39b00
softclock() at softclock+0x79/frame 0xfffffe00d7e39b20
ithread_loop() at ithread_loop+0x24d/frame 0xfffffe00d7e39bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00d7e39bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d7e39bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 8d12h20m43s
Dumping 13931 out of 131002 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55	/usr/src/sys/amd64/include/pcpu_aux.h: No such file or directory.
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:394
#2  0xffffffff80b377eb in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:481
#3  0xffffffff80b37c6a in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:913
#4  0xffffffff80b37a83 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:839
#5  0xffffffff80efab67 in trap_fatal (frame=0xfffffe00d7e39900, eva=16) at /usr/src/sys/amd64/amd64/trap.c:915
#6  0xffffffff80efabbf in trap_pfault (frame=0xfffffe00d7e39900, usermode=<optimized out>, signo=<optimized out>, ucode=<optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:732
#7  0xffffffff80efa221 in trap (frame=0xfffffe00d7e39900) at /usr/src/sys/amd64/amd64/trap.c:398
#8  <signal handler called>
#9  0xffffffff80d2b21c in nd6_llinfo_timer (arg=0xfffff80ad277a800) at /usr/src/sys/netinet6/nd6.c:775
#10 0xffffffff80b53a99 in softclock_call_cc (c=<optimized out>, cc=0xffffffff81878d00 <cc_cpu>, direct=0) at /usr/src/sys/kern/kern_timeout.c:703
#11 0xffffffff80b53ef9 in softclock (arg=0xffffffff81878d00 <cc_cpu>) at /usr/src/sys/kern/kern_timeout.c:823
#12 0xffffffff80af8f8d in intr_event_execute_handlers (p=<optimized out>, ie=0xfffff80101cee600) at /usr/src/sys/kern/kern_intr.c:1168
#13 ithread_execute_handlers (p=<optimized out>, ie=0xfffff80101cee600) at /usr/src/sys/kern/kern_intr.c:1181
#14 ithread_loop (arg=0xfffff80101ce6160) at /usr/src/sys/kern/kern_intr.c:1269
#15 0xffffffff80af5a3e in fork_exit (callout=0xffffffff80af8d40 <ithread_loop>, arg=0xfffff80101ce6160, frame=0xfffffe00d7e39c00)
    at /usr/src/sys/kern/kern_fork.c:1052
#16 <signal handler called>
(kgdb) f 9
#9  0xffffffff80d2b21c in nd6_llinfo_timer (arg=0xfffff80ad277a800) at /usr/src/sys/netinet6/nd6.c:775
775	/usr/src/sys/netinet6/nd6.c: No such file or directory.
(kgdb) i lo
et = {et_link = {tqe_next = 0x0, tqe_prev = 0xfffffe00d7e26bd8}, et_td = 0xfffffe00d3e8d300, et_section = {bucket = 1}}
src = {__u6_addr = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}
ln = 0xfffff80ad277a800
ifp = 0xfffff811058dd800
saved_vnet = <optimized out>
send_ns = <optimized out>
ndi = <optimized out>
pdst = <optimized out>
dst = <optimized out>
do_switch = <optimized out>
psrc = <optimized out>
delay = <optimized out>
(kgdb) p *ifp
$1 = {if_link = {cstqe_next = 0xfffff811058fc000}, if_clones = {le_next = 0xfffff801520dc000, le_prev = 0xfffff811058fc008}, if_groups = {cstqh_first = 0x0, 
    cstqh_last = 0xfffff811058dd818}, if_alloctype = 6 '\006', if_numa_domain = 255 '\377', if_softc = 0xfffff81a34ef5300, if_llsoftc = 0x0, if_l2com = 0x0, 
  if_dname = 0xffffffff81395998 <vlanname> "vlan", if_dunit = 419, if_index = 16, if_index_reserved = 0, if_xname = "vlan419\000\000\000\000\000\000\000\000", 
  if_description = 0x0, if_flags = 2267394, if_drv_flags = 0, if_capabilities = 6817539, if_capenable = 524288, if_linkmib = 0x0, if_linkmiblen = 0, if_refcount = 0, 
  if_type = 135 '\207', if_addrlen = 6 '\006', if_hdrlen = 4 '\004', if_link_state = 0 '\000', if_mtu = 1500, if_metric = 0, if_baudrate = 10000000000, 
  if_hwassist = 0, if_epoch = 51486, if_lastchange = {tv_sec = 1633476486, tv_usec = 61710}, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, 
    ifq_maxlen = 2048, ifq_mtx = {lock_object = {lo_name = 0xfffff811058dd858 "vlan419", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, 
    ifq_drv_head = 0x0, ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 0, altq_type = 0, altq_flags = 0, altq_disc = 0x0, altq_ifp = 0xfffff811058dd800, 
    altq_enqueue = 0x0, altq_dequeue = 0x0, altq_request = 0x0, altq_clfier = 0x0, altq_classify = 0x0, altq_tbr = 0x0, altq_cdnr = 0x0}, if_linktask = {ta_link = {
      stqe_next = 0x0}, ta_pending = 0, ta_priority = 0 '\000', ta_flags = 0 '\000', ta_func = 0xffffffff80c44d00 <do_link_state_change>, 
    ta_context = 0xfffff811058dd800}, if_addmultitask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0 '\000', ta_flags = 0 '\000', 
    ta_func = 0xffffffff80c44f80 <if_siocaddmulti>, ta_context = 0xfffff811058dd800}, if_addr_lock = {lock_object = {lo_name = 0xffffffff80f62755 "if_addr_lock", 
      lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, if_addrhead = {cstqh_first = 0x0, cstqh_last = 0xfffff811058dd9c8}, if_multiaddrs = {
    cstqh_first = 0x0, cstqh_last = 0xfffff811058dd9d8}, if_amcount = 0, if_addr = 0xfffff818b493ab00, if_hw_addr = 0xfffff81d58d89100, 
  if_broadcastaddr = 0xffffffff81394ab0 <etherbroadcastaddr> "\377\377\377\377\377\377", if_afdata_lock = {lock_object = {lo_name = 0xffffffff80fd2876 "if_afdata", 
      lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, if_afdata = {0x0 <repeats 43 times>}, if_afdata_initialized = 0, if_fib = 0, 
  if_vnet = 0xfffff801012da8c0, if_home_vnet = 0xfffff801012da8c0, if_vlantrunk = 0x0, if_bpf = 0xffffffff818c5128 <dead_bpf_if>, if_pcount = 0, if_bridge = 0x0, 
  if_lagg = 0x0, if_pf_kif = 0x0, if_carp = 0x0, if_label = 0x0, if_netmap = 0x0, if_output = 0xffffffff80c4ecb0 <ifdead_output>, 
  if_input = 0xffffffff80c4ecd0 <ifdead_input>, if_bridge_input = 0x0, if_bridge_output = 0x0, if_bridge_linkstate = 0x0, 
  if_start = 0xffffffff80c4ece0 <ifdead_start>, if_ioctl = 0xffffffff80c4ecf0 <ifdead_ioctl>, if_init = 0xffffffff80c5a680 <vlan_init>, 
  if_resolvemulti = 0xffffffff80c4ed00 <ifdead_resolvemulti>, if_qflush = 0xffffffff80c4ed20 <ifdead_qflush>, if_transmit = 0xffffffff80c4ed30 <ifdead_transmit>, 
  if_reassign = 0xffffffff80c50350 <ether_reassign>, if_get_counter = 0xffffffff80c4ed50 <ifdead_get_counter>, 
  if_requestencap = 0xffffffff80c50280 <ether_requestencap>, if_counters = {0xfffffe03abc91140, 0xfffffe03abc91138, 0xfffffe03abc91130, 0xfffffe03abc91128, 
    0xfffffe03abc91120, 0xfffffe03abc91118, 0xfffffe03abc91110, 0xfffffe03abc91108, 0xfffffe03abc91100, 0xfffffe03abc910f8, 0xfffffe03abc910f0, 0xfffffe03abc910e8}, 
  if_hw_tsomax = 65518, if_hw_tsomaxsegcount = 23, if_hw_tsomaxsegsize = 65536, if_snd_tag_alloc = 0xffffffff80c4ed60 <ifdead_snd_tag_alloc>, 
  if_snd_tag_modify = 0xffffffff80c4ed70 <ifdead_snd_tag_modify>, if_snd_tag_query = 0xffffffff80c4ed80 <ifdead_snd_tag_query>, 
  if_snd_tag_free = 0xffffffff80c4ed90 <ifdead_snd_tag_free>, if_ratelimit_query = 0xffffffff80c4eda0 <ifdead_ratelimit_query>, if_ratelimit_setup = 0x0, 
  if_pcp = 0 '\000', if_debugnet_methods = 0x0, if_epoch_ctx = {data = {0xffffffff80c45290 <if_destroy>, 0xfffff801d5acfdc0}}, if_ispare = {0, 0, 0, 0}}
(kgdb) p ifp->if_afdata
$2 = {0x0 <repeats 43 times>}
(kgdb) p ifp->if_afdata[28]
$3 = (void *) 0x0
(kgdb) p ifp->if_flags 
$4 = 2267394
(kgdb) p/x ifp->if_flags 
$5 = 0x229902
(kgdb)

(b) I guess I should go and look where the nd6_llinfo_timer() comes from,,, or let me ask: what drains the callout?
Changing the order to me sounds like hiding a different problem (which likely is a missing callout_drain)?

lltable_free() will stop all LLE related callouts. And in time when we run nd6_ifdetach() LLE timer can not fire.