I ran tests with D22777.65668.diff for some 15 hours before running into this (unrelated ?) deadlock:
https://people.freebsd.org/~pho/stress/log/dougm067.txt
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Dec 15 2019
Dec 14 2019
With D22777.65663.diff I see:
ukbd0: <Keyboard Interface> on usbus2 kbd2 at ukbd0 mountroot: waiting for device /dev/da0p2... panic: vm_map_splay_split: max_free invariant fails cpuid = 9 time = 1576352787 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe012ffbc590 vpanic() at vpanic+0x17e/frame 0xfffffe012ffbc5f0 panic() at panic+0x43/frame 0xfffffe012ffbc650 vm_map_splay() at vm_map_splay+0x3c4/frame 0xfffffe012ffbc750 vm_map_lookup_entry() at vm_map_lookup_entry+0xd1/frame 0xfffffe012ffbc7b0 vm_map_lookup() at vm_map_lookup+0x5b/frame 0xfffffe012ffbc880 vm_fault() at vm_fault+0xba/frame 0xfffffe012ffbca00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe012ffbca40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe012ffbcac0 trap() at trap+0x470/frame 0xfffffe012ffbcbf0 calltrap() at calltrap+0x8/frame 0xfffffe012ffbcbf0 --- trap 0xc, rip = 0x800380f8f, rsp = 0x7fffffffc240, rbp = 0x7fffffffc290 --- KDB: enter: panic [ thread pid 48 tid 100222 ] Stopped at kdb_enter+0x37: movq $0,0x10803b6(%rip) db> x/s version version: FreeBSD 13.0-CURRENT #2 r355746M: Sat Dec 14 20:43:38 CET 2019\012 pho@t2.osted.lan:/usr/src/sys/amd64/compile/PHO\012 db>
With D22777.65647.diff I got:
I ran tests for 20 hours on D22666.65577.diff without seeing any problems.
Dec 13 2019
With D22777.65534.diff I see:
ses0: pass2,cd0 in 'Slot 01', SATA Slot: scbus2 target 0 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub1: 4 ports with 4 removable, self powered panic: map 0xfffff80841282000 max = 7ff7df8e9000, max_left = 0, max_right = 0 cpuid = 10 time = 1576237136 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0130dd5670 vpanic() at vpanic+0x17e/frame 0xfffffe0130dd56d0 panic() at panic+0x43/frame 0xfffffe0130dd5730 _vm_map_assert_consistent() at _vm_map_assert_consistent+0x18c/frame 0xfffffe0130dd5750 vm_map_lookup_entry() at vm_map_lookup_entry+0x589/frame 0xfffffe0130dd57a0 vm_map_lookup() at vm_map_lookup+0xbd/frame 0xfffffe0130dd5880 vm_fault() at vm_fault+0xba/frame 0xfffffe0130dd5a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe0130dd5a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe0130dd5ac0 trap() at trap+0x470/frame 0xfffffe0130dd5bf0 calltrap() at calltrap+0x8/frame 0xfffffe0130dd5bf0 --- trap 0xc, rip = 0x800242355, rsp = 0x7fffffffe1f8, rbp = 0x7fffffffe280 --- KDB: enter: panic [ thread pid 46 tid 100221 ] Stopped at kdb_enter+0x37: movq $0,0x1080086(%rip) db> x/s version version: FreeBSD 13.0-CURRENT #0 r355711M: Fri Dec 13 12:33:37 CET 2019\012 pho@t2.osted.lan:/usr/src/sys/amd64/compile/PHO\012 db>
Dec 8 2019
Dec 2 2019
Ran a brief 5 hour test with D21964.65115.diff. No problems seen.
Dec 1 2019
I completed a full stress2 test of r355146+1d4f3fce228-c245581(cpuctl). No problems seen.
Nov 29 2019
I ran tests for 5 hours on D21964.65043.diff. No problems seen.
Nov 28 2019
I ran a preliminary 3 hours test with 400 test cases. No problems seen.
Nov 26 2019
I ran tests on D22544.64854 for 14 hours. No problems seen.
Nov 25 2019
20191125 08:32:45 all (271/680): proccontrol.sh panic: caller failed to provide space 0x400000 at address 0x800a56000 cpuid = 4 time = 1574667167 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01b74cc6c0 vpanic() at vpanic+0x17e/frame 0xfffffe01b74cc720 panic() at panic+0x43/frame 0xfffffe01b74cc780 vm_map_alignspace() at vm_map_alignspace+0x9b/frame 0xfffffe01b74cc7f0 vm_map_find() at vm_map_find+0x697/frame 0xfffffe01b74cc8b0 vm_map_find_min() at vm_map_find_min+0x85/frame 0xfffffe01b74cc930 vm_mmap_object() at vm_mmap_object+0x395/frame 0xfffffe01b74cc9c0 kern_mmap() at kern_mmap+0x56b/frame 0xfffffe01b74ccaa0 sys_mmap() at sys_mmap+0x2a/frame 0xfffffe01b74ccac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe01b74ccbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe01b74ccbf0 --- syscall (477, FreeBSD ELF64, sys_mmap), rip = 0x800ec3f6a, rsp = 0x7fffff8ea238, rbp = 0x7fffff8ea250 --- KDB: enter: panic [ thread pid 65014 tid 102218 ] Stopped at kdb_enter+0x37: movq $0,0x1082cc6(%rip) db>
Nov 24 2019
I ran the full stress2 test set without finding any problems.
Nov 19 2019
I ran tests with Diff 64526 for 16 hours without seeing any problems.
Nov 18 2019
20191118 12:46:52 all (2/12): sort.sh panic: Lookup object not swappable cpuid = 5 time = 1574077630 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00e3f802a0 vpanic() at vpanic+0x17e/frame 0xfffffe00e3f80300 panic() at panic+0x43/frame 0xfffffe00e3f80360 swp_pager_meta_lookup() at swp_pager_meta_lookup+0x68/frame 0xfffffe00e3f80390 swp_pager_xfer_source() at swp_pager_xfer_source+0x66/frame 0xfffffe00e3f803d0 swp_pager_meta_transfer() at swp_pager_meta_transfer+0x1b1/frame 0xfffffe00e3f80450 swap_pager_copy() at swap_pager_copy+0x267/frame 0xfffffe00e3f80480 vm_object_collapse() at vm_object_collapse+0x178/frame 0xfffffe00e3f804f0 vm_object_deallocate() at vm_object_deallocate+0x4b1/frame 0xfffffe00e3f80540 vm_map_process_deferred() at vm_map_process_deferred+0xad/frame 0xfffffe00e3f80560 vm_map_remove() at vm_map_remove+0xeb/frame 0xfffffe00e3f80590 exec_new_vmspace() at exec_new_vmspace+0x186/frame 0xfffffe00e3f805f0 exec_elf64_imgact() at exec_elf64_imgact+0x82a/frame 0xfffffe00e3f806e0 kern_execve() at kern_execve+0x5e9/frame 0xfffffe00e3f80a40 sys_execve() at sys_execve+0x4c/frame 0xfffffe00e3f80ac0 amd64_syscall() at amd64_syscall+0x2d0/frame 0xfffffe00e3f80bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e3f80bf0 --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x80086675a, rsp = 0x7fffffffe728, rbp = 0x7fffffffe730 ---
Nov 17 2019
I tested D22405.64459.diff for 10 hours without seeing any problems.
Nov 12 2019
I ran the full stress2 test using 64144. No problems seen.
Nov 11 2019
I only saved the vmcore: https://people.freebsd.org/~pho/vmcore.775.xz
Just an update: I'm at test number 534 out of 677 with an uptime of 20 hours (tests are sorted by increasing run time).
No problems so far.
Nov 10 2019
With Diff 64139 I got a "panic: vm_object_madvise: page 0xfffffe0000174cf8 is not managed", which to me seems unrelated.
https://people.freebsd.org/~pho/stress/log/dougm063.txt
I completed a full stress2 test with D22261.64008.diff.
The only problem I saw was this (unrelated) panic: https://people.freebsd.org/~pho/stress/log/mark107.txt
Nov 7 2019
In D22261#486883, @markj wrote:Peter, could you please test this patch?
Nov 6 2019
Nov 2 2019
I ran all of the stress2 tests on Diff 63804 without seeing any problems.
Oct 30 2019
Still doesn't compile.
Oct 28 2019
Oct 27 2019
Serial console on mercat1
(kgdb) l *trap_pfault+0x99 0xffffffff8104d129 is in trap_pfault (../../../amd64/amd64/trap.c:743). 738 * lock, then it is most likely a fatal kernel page fault. 739 * If WITNESS is enabled, then it's going to whine about 740 * bogus LORs with various VM locks, so just skip to the 741 * fatal trap handling directly. 742 */ 743 if (td->td_critnest != 0 || 744 WITNESS_CHECK(WARN_SLEEPOK | WARN_GIANTOK, NULL, 745 "Kernel page fault") != 0) { 746 trap_fatal(frame, eva); 747 return (-1); (kgdb)
This is a KPTI test. This looks unrelated to me, but ...
20191027 15:16:52 all (469/677): kpti.sh Fata lpanic: Assertion p->tp_row < t->t_winsize.tp_row failed at ../../../teken/teken.c:105 cpuid = 10 time = 1572185813 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffff807d872fd60 vpanic() at vpanic+0x19d/frame 0xfffff807d872fdb0 panic() at panic+0x43/frame 0xfffff807d872fe10 teken_subr_do_putchar() at teken_subr_do_putchar+0x138/frame 0xfffff807d872fe60 teken_subr_regular_character() at teken_subr_regular_character+0x27a/frame 0xfffff807d872fea0 teken_input_char() at teken_input_char+0xa7/frame 0xfffff807d872fec0 teken_input() at teken_input+0x111/frame 0xfffff807d872fef0 termcn_cnputc() at termcn_cnputc+0x6e/frame 0xfffff807d872ff20 cnputc() at cnputc+0x7d/frame 0xfffff807d872ff50 cnputs() at cnputs+0x7a/frame 0xfffff807d872ff80 putchar() at putchar+0x150/frame 0xfffff807d8730000 kvprintf() at kvprintf+0x145/frame 0xfffff807d8730120 ??() at 0xffffffff00013051/frame 0xfffff807d87301f0 printf() at printf+0x43/frame 0xfffff807d8730250 trap_fatal() at trap_fatal+0x9c/frame 0xfffff807d87302b0 trap_pfault() at trap_pfault+0x99/frame 0xfffff807d8730310 trap() at trap+0x46f/frame 0xfffff807d8730420 calltrap() at calltrap+0x8/frame 0xfffff807d8730420 --- trap 0x246, rip = 0x80020ac38, rsp = 0x7fffffffdc90, rbp = 0x7fffffffe760 --- KDB: enter: panic [ thread pid 256166277 tid 3224064 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db>
Oct 20 2019
Here's a
Oct 19 2019
I tested 63458 for 17 hours with a "-O0" build of uma_core.c. LGTM.
Oct 18 2019
In D22081#482480, @markj wrote:Peter, could you post the disassembly of uma_vm_zone_stats() from your -O0 kernel?
In D20635#482456, @markj wrote:In D20635#482367, @pho wrote:This too, what not what I was hoping for:
Are you compiling with -O0 or some other unusual compiler flags? I suspect the patch in D22081 is required, though it has no effect on generated code in my testing.
This too, what not what I was hoping for:
20191018 08:58:59 all (284/676): pipe2.sh
Oct 17 2019
Oct 15 2019
With debug.vmmap_check=1 I saw this problem:
https://people.freebsd.org/~pho/stress/log/dougm056.txt
Oct 14 2019
In D21964#481052, @alc wrote:In D21964#480590, @pho wrote:I ran all of the stress2 test on 63130. I found one problem, which to me looks unrelated.
https://people.freebsd.org/~pho/stress/log/quota10-2.txtPeter,
Are you setting debug.vmmap_check to 1?
Oct 12 2019
I ran all of the stress2 test on 63130. I found one problem, which to me looks unrelated.
https://people.freebsd.org/~pho/stress/log/quota10-2.txt
Oct 10 2019
startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "UMA Zones", 3 boot pages left startup_alloc from "UMA Hash", 2 boot pages left startup_alloc from "UMA Zones", 1 boot pages left Entering uma_startup1 with 0 boot pages left panic: map 0xfffff80003002000 max = 7d8df000, max_left = 1ff80000000, max_right = 7d8df000 cpuid = 0 time = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff824ef830 vpanic() at vpanic+0x19d/frame 0xffffffff824ef880 panic() at panic+0x43/frame 0xffffffff824ef8e0 _vm_map_assert_consistent() at _vm_map_assert_consistent+0x3e1/frame 0xffffffff824ef940 vm_map_splay() at vm_map_splay+0x176/frame 0xffffffff824ef9a0 vm_map_lookup_entry() at vm_map_lookup_entry+0xc7/frame 0xffffffff824ef9f0 vm_map_insert() at vm_map_insert+0x1b5/frame 0xffffffff824efae0 kmem_init() at kmem_init+0xab/frame 0xffffffff824efb10 vm_mem_init() at vm_mem_init+0x51/frame 0xffffffff824efb20 mi_startup() at mi_startup+0x210/frame 0xffffffff824efb70 btext() at btext+0x2c KDB: enter: panic [ thread pid 0 tid 0 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db> x/s version version: FreeBSD 13.0-CURRENT #1 r353388M: Thu Oct 10 12:56:20 CEST 2019\012 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 db>
Oct 5 2019
I ran tests on D21882.62898 for more than a day. No problems seen.
Oct 4 2019
Oct 3 2019
I ran tests on D21882.62858.diff for 5 1/2 hours without seeing any problems. I'll move on to 62874.
Sep 27 2019
Sep 6 2019
Tested D20664.61715.diff for 6 hours on amd64. No problems seen.
Sep 4 2019
I ran a brief 10 hour test on amd64 with D20664.61638.diff. No problems seen.
Sep 3 2019
Sep 2 2019
It is still an issue with HEAD:
: : if_delmulti_locked: detaching ifnet instance 0xfffff803d7598000 if_delmulti_locked: detaching ifnet instance 0xfffff803d7598000 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800
Sep 1 2019
I ran the full stress2 test suite with D21471.61489.diff on amd64. No problems seen.
Aug 29 2019
Aug 28 2019
Aug 27 2019
In D21412#466484, @kib wrote:Peter, could you, please, start testing this ?
Aug 25 2019
I ran the full stress2 test with D20814.61081.diff on amd64. LGTM.
I ran all of the stress2 tests with D21357.61095.diff on amd64. LGTM.
Aug 23 2019
I should be able to free up some test hardware later today.
Aug 21 2019
I ran a brief 2 hour test with D21320.61033.diff. No problems seen.
Aug 16 2019
D13671.60851.diff LGTM.
Aug 3 2019
Aug 1 2019
I ran tests for 15 hours on D20814.60344.diff without seeing any problems.
I completed a full stress2 test on amd64 without seeing any problems.
Jul 31 2019
With D20814.60322.diff I got:
20190731 19:27:23 all (1/1): mmap10.sh panic: vm_map_unwire: lookup failed cpuid = 12 time = 1564594061 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c96c5790 vpanic() at vpanic+0x19d/frame 0xfffffe00c96c57e0 panic() at panic+0x43/frame 0xfffffe00c96c5840 vm_map_wire_locked() at vm_map_wire_locked+0x69d/frame 0xfffffe00c96c5900 vm_map_wire() at vm_map_wire+0x40/frame 0xfffffe00c96c5930 kern_mlock() at kern_mlock+0x179/frame 0xfffffe00c96c5980 amd64_syscall() at amd64_syscall+0x2d6/frame 0xfffffe00c96c5ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00c96c5ab0
D20814.60315.diff
20190731 16:18:08 all (17/73): mmap10.sh panic: Lock vm map (user) not exclusively locked @ ../../../vm/vm_map.c:3102 cpuid = 12 time = 1564582693 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c7c80800 vpanic() at vpanic+0x19d/frame 0xfffffe00c7c80850 panic() at panic+0x43/frame 0xfffffe00c7c808b0 _sx_xunlock() at _sx_xunlock+0xab/frame 0xfffffe00c7c808d0 _vm_map_unlock() at _vm_map_unlock+0x75/frame 0xfffffe00c7c80900 vm_map_wire() at vm_map_wire+0x58/frame 0xfffffe00c7c80930 kern_mlock() at kern_mlock+0x179/frame 0xfffffe00c7c80980 amd64_syscall() at amd64_syscall+0x2d6/frame 0xfffffe00c7c80ab0
Jul 30 2019
I ran all of the threaded tests from stress2 on amd64 with D21088.60283.diff.
No problems seen.
Just a follow up. I'm currently testing your patch on amd64 where I have not yet seen any problems.
I got this after a 5 hour test on i386, followed by a shutdown:
panic: vm_fault_hold: fault on nofault entry, addr: 0x83326000 cpuid = 0 time = 1564486041 KDB: stack backtrace: db_trace_self_wrapper(e6666d,1bdcda4,0,23a17534,bd2eb1,...) at db_trace_self_wrapper+0x2a/frame 0x23a17508 kdb_backtrace(e,0,0,2b06100,2db5000,...) at kdb_backtrace+0x2e/frame 0x23a17568 vpanic(163709e,23a175b0,23a175b0,23a17680,12c2044,...) at vpanic+0x121/frame 0x23a17590 panic(163709e,158dad9,83326000,4895800,7000004,...) at panic+0x14/frame 0x23a175a4 vm_fault_hold(2db5000,83326000,2,0,0) at vm_fault_hold+0x2474/frame 0x23a17680 vm_fault(2db5000,83326000,2,0) at vm_fault+0x4a/frame 0x23a176a8 trap_pfault(8332680c) at trap_pfault+0xf3/frame 0x23a176f4 trap(23a177bc,8,28,28,83df8640,...) at trap+0x3c0/frame 0x23a177b0 calltrap() at 0xffc0316d/frame 0x23a177b0 --- trap 0xc, eip = 0xf7f255, esp = 0x23a177fc, ebp = 0x23a17814 --- funsetownlst(27e5ea50) at funsetownlst+0xa5/frame 0x23a17814 pgdelete(27e5ea6c,23c00a80,1651842,36,23a178f8,...) at pgdelete+0x60/frame 0x23a17834 leavepgrp(298d46a8,0,1651842,396,44c,...) at leavepgrp+0xd5/frame 0x23a1784c proc_reap(23c00a80,298d46a8,23a17ba4,36,1edeb50,...) at proc_reap+0x714/frame 0x23a178f8 proc_to_reap(23c00a80,298d46a8,7,0,0,...) at proc_to_reap+0x718/frame 0x23a17998 kern_wait6(23c00a80,7,0,0,23a17ba4,...) at kern_wait6+0x37a/frame 0x23a17a78 kern_wait(23c00a80,ffffffff,23a17ba4,36,0,...) at kern_wait+0x119/frame 0x23a17b80 sys_wait4(23c00a80,23c00d0c) at sys_wait4+0x68/frame 0x23a17c08 syscall(23a17ce8,3b,3b,3b,0,...) at syscall+0x304/frame 0x23a17cdc Xint0x80_syscall() at 0xffc033b7/frame 0x23a17cdc
Ran tests for another 23 hours on D20863.60217.diff. No problems seen.
Jul 27 2019
I tested D20814.60178.diff for 13 hours on amd64 without seeing any problems.
Jul 25 2019
The i386 tests finished with one further problem and an incomplete report: https://people.freebsd.org/~pho/stress/log/alc011.txt
Jul 24 2019
I finished the full stress2 tests on amd64 without any issues.
Jul 22 2019
Sure, happy to.
I ran tests on a GENERICish kernel for 12 hours without any issue, however a kernel build with "options KTR" does not compile:
Jul 21 2019
I still see the panic with D20863.59987.diff:
I ran a brief 4 hour test with this patch.
I completed a full stress2 test with D20947.59944.diff on amd64. No problems seen.
With D20863.59975.diff I got:
Jul 18 2019
A have completed a full stress2 test of D20833.59786.diff without seeing any problems.
Jul 17 2019
I have completed a full stress2 test with D20863.59745.diff on amd64 without seeing any problems.
Jul 15 2019
Some KTR output for a short test of D20863.59745.diff:
https://people.freebsd.org/~pho/stress/log/dougm051.txt
I tested D20949.59742.diff with all of the threaded tests I have, on i386. No problems seen.
Jul 14 2019
With D20863.59728.diff I got:
Jul 13 2019
With D20863.59658.diff I got:
Jul 12 2019
I finished the full stress2 test of D20893.59605.diff without finding any problems.
Jul 11 2019
The test has now run for 12 hours without any problems. I'll let the test run for the full ~48 hours.