kqueue: avoid closing a file under the knlist lock
AcceptedPublic
Actions

Authored by guest-seuros on Wed, Aug 5, 8:37 PM.

Details

Reviewers

kib
markj
kevans
adrian
imp

Summary

Killing a knote releases its file reference, and releasing the last one
runs the close path inline.
panic: _mtx_lock_sleep: recursed on non-recursive mutex ttymtx

Revoking a controlling tty during exit reaches this whenever a knote is
still registered on it.

Released the knlist lock around the drop and restart the walk.
The knote stays valid while the lock is released.

MFC: 1 week

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 75399
Build 72282: arc lint + arc unit

Event Timeline

guest-seuros created this revision.Wed, Aug 5, 8:37 PM

Herald added a subscriber: olce. · View Herald TranscriptWed, Aug 5, 8:37 PM

guest-seuros requested review of this revision.Wed, Aug 5, 8:37 PM

Harbormaster completed remote builds in B75399: Diff 183491.Wed, Aug 5, 8:37 PM

This is difficult to reproduce, but it shows up much more frequently with Rust-based applications.
A common pattern in the Rust ecosystem is aggressive filesystem watching.
Applications such as Starship, Codex, and Grok Build continuously monitor files for changes,
which puts sustained pressure on the kqueue/knote infrastructure.
Left running for several days, they can exhaust the file allocator, making this issue much easier to trigger.

This crash kept happening when i leave starship open for extended time 1/2 day , then emit a poweroff/reboot.

kbowling added a subscriber: kbowling.Thu, Aug 6, 8:01 AM

Can you show the full panic report from the kernel?

I think that the change is fine but not sure what does happen in your case specifically. In particular, if it would be closing of the controlling terminal, it should be reproduced deterministically easier. Did you tried to write a reproducer?

Full panic + backtrace:

panic: _mtx_lock_sleep: recursed on non-recursive mutex ttymtx @ /usr/src/sys/kern/tty.c:377

cpuid = 0
time = 1785959880
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame 0xfffffe00f66c04b0
vpanic() at vpanic+0x149/frame 0xfffffe00f66c05e0
panic() at panic+0x43/frame 0xfffffe00f66c0640
mtx_lock_sleep() at mtx_lock_sleep+0x3b0/frame 0xfffffe00f66c06d0
mtx_lock_flags() at mtx_lock_flags+0x93/frame 0xfffffe00f66c06f0
ttydev_close() at ttydev_close+0x35/frame 0xfffffe00f66c0720
devfs_close() at devfs_close+0x4b0/frame 0xfffffe00f66c0790
VOP_CLOSE_APV() at VOP_CLOSE_APV+0x52/frame 0xfffffe00f66c07c0
vn_close1() at vn_close1+0xef/frame 0xfffffe00f66c0840
vn_closefile() at vn_closefile+0x48/frame 0xfffffe00f66c08c0
devfs_close_f() at devfs_close_f+0x2a/frame 0xfffffe00f66c08f0
_fdrop() at _fdrop+0x1a/frame 0xfffffe00f66c0910
knote_drop_detached() at knote_drop_detached+0x187/frame 0xfffffe00f66c0950

I believe it something wrong with an ecosystem Rust crate , machines without starship did not crashes, also machines that were rebooted below 24h , didn't dump either.

One of the crates might be hammering FreeBSD with linux syscalls.

I tried to reproduce it with code, in vain.
But if i leave machine with limited ram and diskspace, (4gb, 64gb disk), it crashes after 1 day.

It is definitely not the full backtrace.

panic: _mtx_lock_sleep: recursed on non-recursive mutex ttymtx @ /usr/src/sys/kern/tty.c:377

Reading symbols from /boot/kernel.old/kernel...
Reading symbols from /usr/lib/debugboot/kernel.old/kernel.debug...
Reading symbols from /boot/kernel/fdescfs.ko...
Reading symbols from /usr/lib/debugboot/kernel/fdescfs.ko.debug...
Reading symbols from /boot/kernel/firewire.ko...
Reading symbols from /usr/lib/debugboot/kernel/firewire.ko.debug...
Reading symbols from /boot/kernel/fwcam.ko...
Reading symbols from /usr/lib/debugboot/kernel/fwcam.ko.debug...
Reading symbols from /boot/kernel/video.ko...
Reading symbols from /usr/lib/debugboot/kernel/video.ko.debug...
Reading symbols from /boot/kernel/sdhci_pci.ko...
Reading symbols from /usr/lib/debugboot/kernel/sdhci_pci.ko.debug...
Reading symbols from /boot/kernel/tb.ko...
Reading symbols from /usr/lib/debugboot/kernel/tb.ko.debug...
Reading symbols from /boot/kernel/acpi_wmi.ko...
Reading symbols from /usr/lib/debugboot/kernel/acpi_wmi.ko.debug...
Reading symbols from /boot/kernel/mmcsd.ko...
Reading symbols from /usr/lib/debugboot/kernel/mmcsd.ko.debug...
Reading symbols from /boot/modules/i915kms.ko...
(No debugging symbols found in /boot/modules/i915kms.ko)
Reading symbols from /boot/modules/drm.ko...
(No debugging symbols found in /boot/modules/drm.ko)
Reading symbols from /boot/kernel/iic.ko...
Reading symbols from /usr/lib/debugboot/kernel/iic.ko.debug...
Reading symbols from /boot/kernel/linuxkpi_video.ko...
Reading symbols from /usr/lib/debugboot/kernel/linuxkpi_video.ko.debug...
Reading symbols from /boot/modules/dmabuf.ko...
(No debugging symbols found in /boot/modules/dmabuf.ko)
Reading symbols from /boot/kernel/lindebugfs.ko...
Reading symbols from /usr/lib/debugboot/kernel/lindebugfs.ko.debug...
Reading symbols from /boot/modules/ttm.ko...
(No debugging symbols found in /boot/modules/ttm.ko)
Reading symbols from /boot/kernel/linux.ko...
Reading symbols from /usr/lib/debugboot/kernel/linux.ko.debug...
Reading symbols from /boot/kernel/linux_common.ko...
Reading symbols from /usr/lib/debugboot/kernel/linux_common.ko.debug...
Reading symbols from /boot/kernel/linux64.ko...
Reading symbols from /usr/lib/debugboot/kernel/linux64.ko.debug...
Reading symbols from /boot/kernel/pty.ko...
Reading symbols from /usr/lib/debugboot/kernel/pty.ko.debug...
Reading symbols from /boot/kernel/linprocfs.ko...
Reading symbols from /usr/lib/debugboot/kernel/linprocfs.ko.debug...
Reading symbols from /boot/kernel/linsysfs.ko...
Reading symbols from /usr/lib/debugboot/kernel/linsysfs.ko.debug...
Reading symbols from /boot/kernel/hms.ko...
Reading symbols from /usr/lib/debugboot/kernel/hms.ko.debug...
Reading symbols from /boot/kernel/hidmap.ko...
Reading symbols from /usr/lib/debugboot/kernel/hidmap.ko.debug...
Reading symbols from /boot/kernel/ng_socket.ko...
Reading symbols from /usr/lib/debugboot/kernel/ng_socket.ko.debug...
Reading symbols from /boot/kernel/mac_ntpd.ko...
Reading symbols from /usr/lib/debugboot/kernel/mac_ntpd.ko.debug...
curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
57 asm("movq %%gs:%c1,%0" : "=r" (td)
(kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
td = <optimized out>
#1 doadump (textdump=textdump@entry=1)
at /usr/src/sys/kern/kern_shutdown.c:399
    error = 0
    coredump = <optimized out>
#2 0xffffffff809a7467 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:519
    once = 0
    __pc = 0x0
#3 0xffffffff809a797d in vpanic (fmt=<optimized out>,
ap=ap@entry=0xfffffe00f66c0620) at /usr/src/sys/kern/kern_shutdown.c:974
    buf = "_mtx_lock_sleep: recursed on non-recursive mutex ttymtx @ /usr/src/sys/kern/tty.c:377\n", '\000' <repeats 169 times>
    __pc = 0x0
    __pc = 0x0
    __pc = 0x0
    other_cpus = {__bits = {14, 0 <repeats 15 times>}}
    td = 0xfffff8001ec84780
    bootopt = <unavailable>
    newpanic = <optimized out>
#4 0xffffffff809a77c3 in panic (fmt=<unavailable>)
at /usr/src/sys/kern/kern_shutdown.c:887
    ap = {{gp_offset = 32, fp_offset = 48,
        overflow_arg_area = 0xfffffe00f66c0650,
        reg_save_area = 0xfffffe00f66c05f0}}
#5 0xffffffff80986860 in __mtx_lock_sleep (c=0xfffff800034ce420,
v=18446735278132971392, opts=<optimized out>, file=<optimized out>,
line=<optimized out>) at /usr/src/sys/kern/kern_mutex.c:564
    lda = {config = 0x18c0302, delay = 0, spin_cnt = 4294965252}
    sleep_cnt = 0
    sleep_time = 0
    all_time = 0
    doing_lockprof = <optimized out>
    td = 0xfffff8001ec84780
    tid = 18446735278132971392
    m = 0xfffff800034ce408
    owner = <optimized out>
    ts = <optimized out>
#6 0xffffffff80986463 in __mtx_lock_flags (c=<unavailable>,
opts=<unavailable>, opts@entry=0, file=<unavailable>, line=<unavailable>,
line@entry=377) at /usr/src/sys/kern/kern_mutex.c:300
    m = <optimized out>
    tid = 18446735278132971392
    v = <unavailable>
#7 0xffffffff80a3d095 in ttydev_close (dev=0xfffff8001a5f7000, fflag=2097159,
devtype=<optimized out>, td=0xfffff8001ec84780)
at /usr/src/sys/kern/tty.c:377
    tp = 0xfffff800034ce400
#8 0xffffffff80881510 in devfs_close (ap=0xfffffe00f66c07e8)
at /usr/src/sys/fs/devfs/devfs_vnops.c:780
    ref = 1
    vp = 0xfffff802e495c6e0
    td = 0xfffff8001ec84780
    de = <optimized out>
    dev = 0xfffff8001a5f7000
    p = <optimized out>
    oldvp = <optimized out>
    dsw = 0xffffffff814af198 <ttydev_cdevsw>
    dflags = <optimized out>
    vp_locked = 2097152
    error = <optimized out>
#9 0xffffffff80ea5962 in VOP_CLOSE_APV (
vop=0xffffffff8147c1d0 <devfs_specops>, a=a@entry=0xfffffe00f66c07e8)
at vnode_if.c:469
    rc = <optimized out>
#10 0xffffffff80ac73af in VOP_CLOSE (vp=0xfffff802e495c6e0, fflag=3,
cred=0xfffff8016a9f2780, td=0xfffff8001ec84780) at ./vnode_if.h:290
    a = {a_gen = {a_desc = 0xffffffff81512428 <vop_close_desc>},
      a_vp = 0xfffff802e495c6e0, a_fflag = 3, a_cred = 0xfffff8016a9f2780,
      a_td = 0xfffff8001ec84780}
#11 vn_close1 (vp=vp@entry=0xfffff802e495c6e0, flags=3,
file_cred=0xfffff8016a9f2780, td=0xfffff8001ec84780, keep_ref=false)
at /usr/src/sys/kern/vfs_vnops.c:588
    mp = 0xfffffe00c2611100
    lock_flags = 2098176
    error = <optimized out>
#12 0xffffffff80ac5228 in vn_closefile (fp=0xfffff8001ed41d70,
td=<unavailable>) at /usr/src/sys/kern/vfs_vnops.c:2061
    lf = {l_start = 83952129, l_len = 83952129, l_pid = -160692064,
      l_type = -512, l_whence = -1, l_sysid = -2133468456}
    vp = 0xfffff802e495c6e0
    ref = <unavailable>
    error = <optimized out>
#13 0xffffffff80881f8a in devfs_close_f (fp=<unavailable>,
fp@entry=<error reading variable: value is not available>,
td=<unavailable>,
td@entry=<error reading variable: value is not available>)
at /usr/src/sys/fs/devfs/devfs_vnops.c:799
    fpop = 0x0
    error = <optimized out>
#14 0xffffffff8093ee3a in fo_close (fp=0xfffff8001ed41d70, td=<unavailable>)
at /usr/src/sys/sys/file.h:413
No locals.
#15 _fdrop (fp=0xfffff8001ed41d70, td=<unavailable>,
td@entry=0xfffff8001ec84780) at /usr/src/sys/kern/kern_descrip.c:3954
    error = <optimized out>
#16 0xffffffff8094f007 in knote_drop_detached (kn=kn@entry=0xfffff802e48bf280,
td=td@entry=0xfffff8001ec84780) at /usr/src/sys/kern/kern_event.c:2966
    _error = 0
    _fp = <unavailable>
    kq = 0xfffff8044a565700
    list = <optimized out>
#17 0xffffffff8094ed3a in knlist_cleardel (knl=0xfffff800034ce510,
td=0xfffff8001ec84780, islocked=1, killkn=1)
at /usr/src/sys/kern/kern_event.c:2823
    kn = 0xfffff802e48bf280
    kn2 = 0x0
    kq = <optimized out>
#18 0xffffffff80a3d115 in ttydev_close (dev=0xfffff8001a5f7000, fflag=2097156,
devtype=<optimized out>, td=0xfffff8001ec84780)
at /usr/src/sys/kern/tty.c:398
    tp = 0xfffff800034ce400
#19 0xffffffff80881510 in devfs_close (ap=0xfffffe00f66c0a98)
at /usr/src/sys/fs/devfs/devfs_vnops.c:780
    ref = 1
    vp = 0xfffff802e495c6e0
    td = 0xfffff8001ec84780
    de = <optimized out>
    dev = 0xfffff8001a5f7000
    p = <optimized out>
    oldvp = <optimized out>
    dsw = 0xffffffff814af198 <ttydev_cdevsw>
    dflags = <optimized out>
    vp_locked = 524288
    error = <optimized out>
#20 0xffffffff80ea5962 in VOP_CLOSE_APV (
vop=0xffffffff8147c1d0 <devfs_specops>, a=a@entry=0xfffffe00f66c0a98)
at vnode_if.c:469
    rc = <optimized out>
#21 0xffffffff80aac364 in VOP_CLOSE (vp=0xfffff802e495c6e0, fflag=4, cred=0x0,
td=0xfffff8001ec84780) at ./vnode_if.h:290
    a = {a_gen = {a_desc = 0xffffffff81512428 <vop_close_desc>},
      a_vp = 0xfffff802e495c6e0, a_fflag = 4, a_cred = 0x0,
      a_td = 0xfffff8001ec84780}
#22 vgonel (vp=vp@entry=0xfffff802e495c6e0)
at /usr/src/sys/kern/vfs_subr.c:4458
    mp = 0xfffffe00c27885d8
    td = 0xfffff8001ec84780
    oweinact = <optimized out>
    object = <optimized out>
    active = <optimized out>
    doinginact = <optimized out>
#23 0xffffffff80aacd6f in vgone (vp=vp@entry=0xfffff802e495c6e0)
at /usr/src/sys/kern/vfs_subr.c:4349
No locals.
#24 0xffffffff8087fe3c in devfs_revoke (ap=<optimized out>)
at /usr/src/sys/fs/devfs/devfs_vnops.c:1686
    vp = 0xfffff802e495c6e0
    dev = 0xfffff8001a5f7000
    cdp = 0xfffff8001a5f7000
    vp2 = <optimized out>
    i = <optimized out>
    de = <optimized out>
    vs = <optimized out>
    loop = <optimized out>
#25 0xffffffff80ea6f31 in VOP_REVOKE_APV (
vop=0xffffffff8147c1d0 <devfs_specops>, a=a@entry=0xfffffe00f66c0ba8)
at vnode_if.c:1319
    rc = <optimized out>
#26 0xffffffff8098d055 in VOP_REVOKE (vp=0xfffff802e495c6e0, flags=1)
at ./vnode_if.h:868
    a = {a_gen = {a_desc = 0xffffffff815133a0 <vop_revoke_desc>},
      a_vp = 0xfffff802e495c6e0, a_flags = 1}
#27 killjobc () at /usr/src/sys/kern/kern_proc.c:959
p = 0xfffffe00c2730038
sp = 0xfffff802e46d0c00
ttyvp = 0xfffff802e495c6e0
tp = 0xfffff800034ce400
#28 0xffffffff80957855 in exit1 (td=td@entry=0xfffff8001ec84780,
rval=rval@entry=0, signo=signo@entry=1)
at /usr/src/sys/kern/kern_exit.c:534
    p = 0xfffffe00c2730038
    q = <optimized out>
    signal_parent = <optimized out>
    nq = <optimized out>
    ksi = <optimized out>
    ksi1 = <optimized out>
    t = <optimized out>
    tdt = <optimized out>
#29 0xffffffff8095712d in kern_exit (td=0xfffff8001ec84780, rval=0, signo=1)
at /usr/src/sys/kern/kern_exit.c:260
    p = 0xfffffe00c2730038
#30 0xffffffff809c8893 in sigexit (td=td@entry=0xfffff8001ec84780,
sig=sig@entry=1) at /usr/src/sys/kern/kern_ucoredump.c:201
    p = <optimized out>
    rv = <optimized out>
    logexit = <optimized out>
#31 0xffffffff809ac3f2 in postsig (sig=1) at /usr/src/sys/kern/kern_sig.c:3478
ksi = {ksi_link = {tqe_next = 0x0, tqe_prev = 0x0}, ksi_info = {
    si_signo = 1, si_errno = 0, si_code = 65537, si_pid = 3977,
    si_uid = 1001, si_status = 0, si_addr = 0x0, si_value = {
      sival_int = 0, sival_ptr = 0x0, sigval_int = 0,
      sigval_ptr = 0x0}, _reason = {_fault = {_trapno = 0}, _timer = {
        _timerid = 0, _overrun = 0}, _mesgq = {_mqd = 0}, _poll = {
        _band = 0}, _capsicum = {_syscall = 0}, __spare__ = {
        __spare1__ = 0, __spare2__ = {0, 0, 0, 0, 0, 0, 0}}}},
  ksi_flags = 0, ksi_sigq = 0x0}
returnmask = {__bits = {1, 0, 0, 0}}
td = 0xfffff8001ec84780
p = 0xfffffe00c2730038
ps = 0xfffff802a197a000
action = 0x0
#32 0xffffffff809b0d9b in ast_sig (td=0xfffff8001ec84780,
td@entry=<error reading variable: value is not available>,
tda=<unavailable>,
tda@entry=<error reading variable: value is not available>)
at /usr/src/sys/kern/kern_sig.c:314
    p = 0xfffffe00c2730038
    old_boundary = <optimized out>
    resched_sigs = <optimized out>
    sig = <optimized out>
#33 0xffffffff80a14cf8 in ast_handler (td=td@entry=0xfffff8001ec84780,
framep=framep@entry=0xfffffe00f66c0f40, dtor=<optimized out>)
at /usr/src/sys/kern/subr_trap.c:350
    td_ast = 32768
    a = 15
    f = <unavailable>
    run = <optimized out>
    ae = <optimized out>
#34 0xffffffff80a14c20 in ast (framep=0xfffffe00f66c0f40)
at /usr/src/sys/kern/subr_trap.c:360
    td = 0xfffff8001ec84780
#35 <signal handler called>
No locals.
#36 0x00000008256eb1ca in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x820523438
(kgdb) Tracing command "bash", '\000' <repeats 15 times> pid 3977 tid 101913 (CPU 0)
#0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
#1 doadump (textdump=textdump@entry=1)
at /usr/src/sys/kern/kern_shutdown.c:399
#2 0xffffffff809a7467 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:519
#3 0xffffffff809a797d in vpanic (fmt=<optimized out>,
ap=ap@entry=0xfffffe00f66c0620) at /usr/src/sys/kern/kern_shutdown.c:974
#4 0xffffffff809a77c3 in panic (fmt=<unavailable>)
at /usr/src/sys/kern/kern_shutdown.c:887
#5 0xffffffff80986860 in __mtx_lock_sleep (c=0xfffff800034ce420,
v=18446735278132971392, opts=<optimized out>, file=<optimized out>,
line=<optimized out>) at /usr/src/sys/kern/kern_mutex.c:564
#6 0xffffffff80986463 in __mtx_lock_flags (c=<unavailable>,
opts=<unavailable>, opts@entry=0, file=<unavailable>, line=<unavailable>,
line@entry=377) at /usr/src/sys/kern/kern_mutex.c:300
#7 0xffffffff80a3d095 in ttydev_close (dev=0xfffff8001a5f7000, fflag=2097159,
devtype=<optimized out>, td=0xfffff8001ec84780)
at /usr/src/sys/kern/tty.c:377
#8 0xffffffff80881510 in devfs_close (ap=0xfffffe00f66c07e8)
at /usr/src/sys/fs/devfs/devfs_vnops.c:780
#9 0xffffffff80ea5962 in VOP_CLOSE_APV (
vop=0xffffffff8147c1d0 <devfs_specops>, a=a@entry=0xfffffe00f66c07e8)
at vnode_if.c:469
#10 0xffffffff80ac73af in VOP_CLOSE (vp=0xfffff802e495c6e0, fflag=3,
cred=0xfffff8016a9f2780, td=0xfffff8001ec84780) at ./vnode_if.h:290
#11 vn_close1 (vp=vp@entry=0xfffff802e495c6e0, flags=3,
file_cred=0xfffff8016a9f2780, td=0xfffff8001ec84780, keep_ref=false)
at /usr/src/sys/kern/vfs_vnops.c:588
#12 0xffffffff80ac5228 in vn_closefile (fp=0xfffff8001ed41d70,
td=<unavailable>) at /usr/src/sys/kern/vfs_vnops.c:2061
#13 0xffffffff80881f8a in devfs_close_f (fp=<unavailable>,
fp@entry=<error reading variable: value is not available>,
td=<unavailable>,
td@entry=<error reading variable: value is not available>)
at /usr/src/sys/fs/devfs/devfs_vnops.c:799
#14 0xffffffff8093ee3a in fo_close (fp=0xfffff8001ed41d70, td=<unavailable>)
at /usr/src/sys/sys/file.h:413
#15 _fdrop (fp=0xfffff8001ed41d70, td=<unavailable>,
td@entry=0xfffff8001ec84780) at /usr/src/sys/kern/kern_descrip.c:3954
#16 0xffffffff8094f007 in knote_drop_detached (kn=kn@entry=0xfffff802e48bf280,
td=td@entry=0xfffff8001ec84780) at /usr/src/sys/kern/kern_event.c:2966
#17 0xffffffff8094ed3a in knlist_cleardel (knl=0xfffff800034ce510,
td=0xfffff8001ec84780, islocked=1, killkn=1)
at /usr/src/sys/kern/kern_event.c:2823
#18 0xffffffff80a3d115 in ttydev_close (dev=0xfffff8001a5f7000, fflag=2097156,
devtype=<optimized out>, td=0xfffff8001ec84780)
at /usr/src/sys/kern/tty.c:398
#19 0xffffffff80881510 in devfs_close (ap=0xfffffe00f66c0a98)
at /usr/src/sys/fs/devfs/devfs_vnops.c:780
#20 0xffffffff80ea5962 in VOP_CLOSE_APV (
vop=0xffffffff8147c1d0 <devfs_specops>, a=a@entry=0xfffffe00f66c0a98)
at vnode_if.c:469
#21 0xffffffff80aac364 in VOP_CLOSE (vp=0xfffff802e495c6e0, fflag=4, cred=0x0,
td=0xfffff8001ec84780) at ./vnode_if.h:290
#22 vgonel (vp=vp@entry=0xfffff802e495c6e0)
at /usr/src/sys/kern/vfs_subr.c:4458
#23 0xffffffff80aacd6f in vgone (vp=vp@entry=0xfffff802e495c6e0)
at /usr/src/sys/kern/vfs_subr.c:4349
#24 0xffffffff8087fe3c in devfs_revoke (ap=<optimized out>)
at /usr/src/sys/fs/devfs/devfs_vnops.c:1686
#25 0xffffffff80ea6f31 in VOP_REVOKE_APV (
vop=0xffffffff8147c1d0 <devfs_specops>, a=a@entry=0xfffffe00f66c0ba8)
at vnode_if.c:1319
#26 0xffffffff8098d055 in VOP_REVOKE (vp=0xfffff802e495c6e0, flags=1)
at ./vnode_if.h:868
#27 killjobc () at /usr/src/sys/kern/kern_proc.c:959
#28 0xffffffff80957855 in exit1 (td=td@entry=0xfffff8001ec84780,
rval=rval@entry=0, signo=signo@entry=1)
at /usr/src/sys/kern/kern_exit.c:534
#29 0xffffffff8095712d in kern_exit (td=0xfffff8001ec84780, rval=0, signo=1)
at /usr/src/sys/kern/kern_exit.c:260
#30 0xffffffff809c8893 in sigexit (td=td@entry=0xfffff8001ec84780,
sig=sig@entry=1) at /usr/src/sys/kern/kern_ucoredump.c:201
#31 0xffffffff809ac3f2 in postsig (sig=1) at /usr/src/sys/kern/kern_sig.c:3478
#32 0xffffffff809b0d9b in ast_sig (td=0xfffff8001ec84780,
td@entry=<error reading variable: value is not available>,
tda=<unavailable>,
tda@entry=<error reading variable: value is not available>)
at /usr/src/sys/kern/kern_sig.c:314
#33 0xffffffff80a14cf8 in ast_handler (td=td@entry=0xfffff8001ec84780,
framep=framep@entry=0xfffffe00f66c0f40, dtor=<optimized out>)
at /usr/src/sys/kern/subr_trap.c:350
#34 0xffffffff80a14c20 in ast (framep=0xfffffe00f66c0f40)
at /usr/src/sys/kern/subr_trap.c:360
#35 <signal handler called>
#36 0x00000008256eb1ca in ?? ()
Backtrace stopped: Cannot access memory at address 0x820523438

Tracing command "starship-rs", '\000' <repeats 14 times> pid 48672 tid 104911 (CPU 1)
#0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1537
#1 0xffffffff80dcd898 in ipi_nmi_handler ()
at /usr/src/sys/x86/x86/mp_x86.c:1494
#2 0xffffffff80dc244f in nmi_handle_intr (frame=<unavailable>)
at /usr/src/sys/x86/x86/cpu_machdep.c:1086
#3 <signal handler called>
#4 _lockmgr_assert (lk=0xfffff803323d8598, what=1, line=1264,
file=<optimized out>) at /usr/src/sys/kern/kern_lock.c:1733
#5 lockmgr_unlock (lk=0xfffff803323d8598)
at /usr/src/sys/kern/kern_lock.c:1266
#6 0xffffffff80ea8512 in VOP_UNLOCK_APV (
vop=0xffffffff81412898 <zfs_vnodeops>, a=a@entry=0xfffffe00f608f888)
at vnode_if.c:2154
#7 0xffffffff80aad6dd in VOP_UNLOCK (vp=<optimized out>) at ./vnode_if.h:1356
#8 vfs_knlunlock (arg=<optimized out>) at /usr/src/sys/kern/vfs_subr.c:6694
#9 0xffffffff8094e93d in kn_list_unlock (knl=0xfffff8043ffbef40)
at /usr/src/sys/kern/kern_event.c:301
#10 knlist_remove_kq (knl=0xfffff8043ffbef40, kn=0xfffff8043fcdc960,
knlislocked=knlislocked@entry=0, kqislocked=kqislocked@entry=0)
at /usr/src/sys/kern/kern_event.c:2666
#11 0xffffffff8094e7b9 in knlist_remove (knl=0xfffff803323d8598, kn=0x80000,
islocked=516247552, islocked@entry=0)
at /usr/src/sys/kern/kern_event.c:2681
#12 0xffffffff80ab83f4 in filt_vfsdetach (kn=0x80000)
at /usr/src/sys/kern/vfs_subr.c:6758
#13 0xffffffff8094dfb7 in knote_drop (kn=0xfffff8043fcdc960,
td=0xfffff8001ec55000) at /usr/src/sys/kern/kern_event.c:2928
#14 kqueue_drain (kq=kq@entry=0xfffff8002c7f7d00,
td=td@entry=0xfffff8001ec55000) at /usr/src/sys/kern/kern_event.c:2437
#15 0xffffffff8094f900 in kqueue_close (fp=0xfffff802a0317370,
td=0xfffff8001ec55000) at /usr/src/sys/kern/kern_event.c:2500
#16 0xffffffff8093ee3a in fo_close (fp=0xfffff802a0317370, td=0x80000)
at /usr/src/sys/sys/file.h:413
#17 _fdrop (fp=0xfffff802a0317370, td=0x80000, td@entry=0xfffff8001ec55000)
at /usr/src/sys/kern/kern_descrip.c:3954
#18 0xffffffff8094f007 in knote_drop_detached (kn=kn@entry=0xfffff80386095c80,
td=td@entry=0xfffff8001ec55000) at /usr/src/sys/kern/kern_event.c:2966
#19 0xffffffff8094dedb in knote_drop (kn=0xfffff80386095c80,
td=0xfffff8001ec55000) at /usr/src/sys/kern/kern_event.c:2929
#20 kqueue_drain (kq=kq@entry=0xfffff8044a564500,
td=td@entry=0xfffff8001ec55000) at /usr/src/sys/kern/kern_event.c:2437
#21 0xffffffff8094f900 in kqueue_close (fp=0xfffff802a031cf50,
td=0xfffff8001ec55000) at /usr/src/sys/kern/kern_event.c:2500
#22 0xffffffff8093ee3a in fo_close (fp=0xfffff802a031cf50, td=0x80000)
at /usr/src/sys/sys/file.h:413
#23 _fdrop (fp=fp@entry=0xfffff802a031cf50, td=0x80000,
td@entry=0xfffff8001ec55000) at /usr/src/sys/kern/kern_descrip.c:3954
#24 0xffffffff80942883 in closef (fp=fp@entry=0xfffff802a031cf50,
td=td@entry=0xfffff8001ec55000) at /usr/src/sys/kern/kern_descrip.c:3032
#25 0xffffffff80941e51 in fdescfree_fds (td=0xfffff8001ec55000,
fdp=0xfffffe0140542920) at /usr/src/sys/kern/kern_descrip.c:2749
#26 fdescfree (td=td@entry=0xfffff8001ec55000)
at /usr/src/sys/kern/kern_descrip.c:2792
#27 0xffffffff8095763d in exit1 (td=td@entry=0xfffff8001ec55000,
rval=rval@entry=0, signo=signo@entry=15)
at /usr/src/sys/kern/kern_exit.c:449
#28 0xffffffff8095712d in kern_exit (td=0xfffff8001ec55000, rval=0, signo=15)
at /usr/src/sys/kern/kern_exit.c:260
#29 0xffffffff809c8893 in sigexit (td=td@entry=0xfffff8001ec55000,
sig=sig@entry=15) at /usr/src/sys/kern/kern_ucoredump.c:201
#30 0xffffffff809ac3f2 in postsig (sig=15) at /usr/src/sys/kern/kern_sig.c:3478
#31 0xffffffff809b0d9b in ast_sig (td=0xfffff8001ec55000, tda=<optimized out>)
at /usr/src/sys/kern/kern_sig.c:314
#32 0xffffffff80a14cf8 in ast_handler (td=td@entry=0xfffff8001ec55000,
framep=framep@entry=0xfffffe00f608ff40, dtor=<optimized out>)
at /usr/src/sys/kern/subr_trap.c:350
#33 0xffffffff80a14c20 in ast (framep=0xfffffe00f608ff40)
at /usr/src/sys/kern/subr_trap.c:360
#34 0xffffffff80dd9926 in doreti_ast ()
at /usr/src/sys/amd64/amd64/exception.S:1132
#35 0x00004d633905c810 in ?? ()
#36 0x0000000000000002 in ?? ()
#37 0x0000000000019dcf in ?? ()
#38 0x0000000000000000 in ?? ()

Tracing command "kernel", '\000' <repeats 13 times> pid 0 tid 100214 (CPU 2)
#0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1537
#1 0xffffffff80dcd898 in ipi_nmi_handler ()
at /usr/src/sys/x86/x86/mp_x86.c:1494
#2 0xffffffff80dc244f in nmi_handle_intr (frame=<unavailable>)
at /usr/src/sys/x86/x86/cpu_machdep.c:1086
#3 <signal handler called>
#4 0xffffffff804b4847 in LZ4_compressCtx (ctx=0xfffffe013ec79000,
source=0xfffffe012a8a0000 <error: Cannot access memory at address 0xfffffe012a8a0000>,
dest=0xfffffe013abcd004 <error: Cannot access memory at address 0xfffffe013abcd004>, isize=131072, osize=114684)
at /usr/src/sys/contrib/openzfs/module/zfs/lz4_zfs.c:540
#5 real_LZ4_compress (
source=0xfffffe012a8a0000 <error: Cannot access memory at address 0xfffffe012a8a0000>,
dest=0xfffffe013abcd004 <error: Cannot access memory at address 0xfffffe013abcd004>, isize=131072, osize=114684)
at /usr/src/sys/contrib/openzfs/module/zfs/lz4_zfs.c:871
#6 zfs_lz4_compress_buf (s_start=0xfffffe012a8a0000,
d_start=0xfffffe013abcd000, s_len=131072, d_len=114688, n=<optimized out>)
at /usr/src/sys/contrib/openzfs/module/zfs/lz4_zfs.c:67
#7 zfs_lz4_compress (src=src@entry=0xfffff80372d58700,
dst=0xfffff8016df23400, s_len=s_len@entry=131072,
d_len=d_len@entry=114688, n=<optimized out>)
at /usr/src/sys/contrib/openzfs/module/zfs/lz4_zfs.c:112
#8 0xffffffff805a712b in zio_compress_data (c=c@entry=ZIO_COMPRESS_LZ4,
src=0xfffff80372d58700, dst=dst@entry=0xfffffe00d3e86dc8,
s_len=s_len@entry=131072, d_len=114688, level=<optimized out>)
at /usr/src/sys/contrib/openzfs/module/zfs/zio_compress.c:146
#9 0xffffffff8059ab16 in zio_write_compress (zio=0xfffffe013762e200)
at /usr/src/sys/contrib/openzfs/module/zfs/zio.c:1975
#10 0xffffffff80597828 in __zio_execute (zio=<optimized out>)
at /usr/src/sys/contrib/openzfs/module/zfs/zio.c:2478
#11 zio_execute (zio=<optimized out>)
at /usr/src/sys/contrib/openzfs/module/zfs/zio.c:2389
#12 0xffffffff80a1214b in taskqueue_run_locked (
queue=queue@entry=0xfffff80001746500)
at /usr/src/sys/kern/subr_taskqueue.c:531
#13 0xffffffff80a130c3 in taskqueue_thread_loop (
arg=arg@entry=0xfffff80001716790) at /usr/src/sys/kern/subr_taskqueue.c:847
#14 0xffffffff8095d952 in fork_exit (
callout=0xffffffff80a12ff0 <taskqueue_thread_loop>,
arg=0xfffff80001716790, frame=0xfffffe00d3e86f40)
at /usr/src/sys/kern/kern_fork.c:1220
#15 <signal handler called>
#16 0xfb885dd05beab9c3 in ?? ()
Backtrace stopped: Cannot access memory at address 0xf73ffffff7bf3f7f

Tracing command "kernel", '\000' <repeats 13 times> pid 0 tid 100212 (CPU 3)
#0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1537
#1 0xffffffff80dcd898 in ipi_nmi_handler ()
at /usr/src/sys/x86/x86/mp_x86.c:1494
#2 0xffffffff80dc244f in nmi_handle_intr (frame=<unavailable>)
at /usr/src/sys/x86/x86/cpu_machdep.c:1086
#3 <signal handler called>
#4 LZ4_NbCommonBytes (val=4003473775495479296, val@entry=12288)
at /usr/src/sys/contrib/openzfs/module/zfs/lz4_zfs.c:428
#5 LZ4_compress64kCtx (ctx=0xfffffe0131ad9000, source=0xfffffe00f74bc000 "",
dest=0xfffffe00fc7b4004 <error: Cannot access memory at address 0xfffffe00fc7b4004>, isize=16384, osize=12284)
at /usr/src/sys/contrib/openzfs/module/zfs/lz4_zfs.c:770
#6 real_LZ4_compress (source=0xfffffe00f74bc000 "",
dest=0xfffffe00fc7b4004 <error: Cannot access memory at address 0xfffffe00fc7b4004>, isize=16384, osize=12284)
at /usr/src/sys/contrib/openzfs/module/zfs/lz4_zfs.c:869
#7 zfs_lz4_compress_buf (s_start=0xfffffe00f74bc000,
d_start=0xfffffe00fc7b4000, s_len=16384, d_len=12288, n=<optimized out>)
at /usr/src/sys/contrib/openzfs/module/zfs/lz4_zfs.c:67
#8 zfs_lz4_compress (src=src@entry=0xfffff803c2027b00,
dst=0xfffff8028b41a700, s_len=s_len@entry=16384, d_len=d_len@entry=12288,
n=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/lz4_zfs.c:112
#9 0xffffffff805a712b in zio_compress_data (c=c@entry=ZIO_COMPRESS_LZ4,
src=0xfffff803c2027b00, dst=dst@entry=0xfffffe00d3e40dc8,
s_len=s_len@entry=16384, d_len=12288, level=<optimized out>)
at /usr/src/sys/contrib/openzfs/module/zfs/zio_compress.c:146
#10 0xffffffff8059ab16 in zio_write_compress (zio=0xfffffe01042b3480)
at /usr/src/sys/contrib/openzfs/module/zfs/zio.c:1975
#11 0xffffffff80597828 in __zio_execute (zio=<optimized out>)
at /usr/src/sys/contrib/openzfs/module/zfs/zio.c:2478
#12 zio_execute (zio=<optimized out>)
at /usr/src/sys/contrib/openzfs/module/zfs/zio.c:2389
#13 0xffffffff80a1214b in taskqueue_run_locked (
queue=queue@entry=0xfffff80001746500)
at /usr/src/sys/kern/subr_taskqueue.c:531
#14 0xffffffff80a130c3 in taskqueue_thread_loop (
arg=arg@entry=0xfffff80001716790) at /usr/src/sys/kern/subr_taskqueue.c:847
#15 0xffffffff8095d952 in fork_exit (
callout=0xffffffff80a12ff0 <taskqueue_thread_loop>,
arg=0xfffff80001716790, frame=0xfffffe00d3e40f40)
at /usr/src/sys/kern/kern_fork.c:1220
#16 <signal handler called>
#17 0x929dd515a0bd0019 in ?? ()
Backtrace stopped: Cannot access memory at address 0xeffff5ffdfffeeff

(kgdb)

So the lock recursion is only a symptom. The problem is that we recurse into the ttydev_close(). I believe this is what should be fixed. See D58706.
This does not mean that your change is wrong, I do think that we should not call fdrop() on file with a lock held, on principle.

This revision is now accepted and ready to land.Fri, Aug 7, 7:58 PM

Revision Contents
Changeset List

Path

Size

sys/

kern/

kern_event.c

8 lines

Diff 183491

View Options

kqueue: avoid closing a file under the knlist lockAcceptedPublicActions