Here's a
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Oct 20 2019
Oct 19 2019
I tested 63458 for 17 hours with a "-O0" build of uma_core.c. LGTM.
Oct 18 2019
In D22081#482480, @markj wrote:Peter, could you post the disassembly of uma_vm_zone_stats() from your -O0 kernel?
In D20635#482456, @markj wrote:In D20635#482367, @pho wrote:This too, what not what I was hoping for:
Are you compiling with -O0 or some other unusual compiler flags? I suspect the patch in D22081 is required, though it has no effect on generated code in my testing.
This too, what not what I was hoping for:
20191018 08:58:59 all (284/676): pipe2.sh
Oct 17 2019
Oct 15 2019
With debug.vmmap_check=1 I saw this problem:
https://people.freebsd.org/~pho/stress/log/dougm056.txt
Oct 14 2019
In D21964#481052, @alc wrote:In D21964#480590, @pho wrote:I ran all of the stress2 test on 63130. I found one problem, which to me looks unrelated.
https://people.freebsd.org/~pho/stress/log/quota10-2.txtPeter,
Are you setting debug.vmmap_check to 1?
Oct 12 2019
I ran all of the stress2 test on 63130. I found one problem, which to me looks unrelated.
https://people.freebsd.org/~pho/stress/log/quota10-2.txt
Oct 10 2019
startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "UMA Zones", 3 boot pages left startup_alloc from "UMA Hash", 2 boot pages left startup_alloc from "UMA Zones", 1 boot pages left Entering uma_startup1 with 0 boot pages left panic: map 0xfffff80003002000 max = 7d8df000, max_left = 1ff80000000, max_right = 7d8df000 cpuid = 0 time = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff824ef830 vpanic() at vpanic+0x19d/frame 0xffffffff824ef880 panic() at panic+0x43/frame 0xffffffff824ef8e0 _vm_map_assert_consistent() at _vm_map_assert_consistent+0x3e1/frame 0xffffffff824ef940 vm_map_splay() at vm_map_splay+0x176/frame 0xffffffff824ef9a0 vm_map_lookup_entry() at vm_map_lookup_entry+0xc7/frame 0xffffffff824ef9f0 vm_map_insert() at vm_map_insert+0x1b5/frame 0xffffffff824efae0 kmem_init() at kmem_init+0xab/frame 0xffffffff824efb10 vm_mem_init() at vm_mem_init+0x51/frame 0xffffffff824efb20 mi_startup() at mi_startup+0x210/frame 0xffffffff824efb70 btext() at btext+0x2c KDB: enter: panic [ thread pid 0 tid 0 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db> x/s version version: FreeBSD 13.0-CURRENT #1 r353388M: Thu Oct 10 12:56:20 CEST 2019\012 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 db>
Oct 5 2019
I ran tests on D21882.62898 for more than a day. No problems seen.
Oct 4 2019
Oct 3 2019
I ran tests on D21882.62858.diff for 5 1/2 hours without seeing any problems. I'll move on to 62874.
Sep 27 2019
Sep 6 2019
Tested D20664.61715.diff for 6 hours on amd64. No problems seen.
Sep 4 2019
I ran a brief 10 hour test on amd64 with D20664.61638.diff. No problems seen.
Sep 3 2019
Sep 2 2019
It is still an issue with HEAD:
: : if_delmulti_locked: detaching ifnet instance 0xfffff803d7598000 if_delmulti_locked: detaching ifnet instance 0xfffff803d7598000 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800 if_delmulti_locked: detaching ifnet instance 0xfffff803bc652800
Sep 1 2019
I ran the full stress2 test suite with D21471.61489.diff on amd64. No problems seen.
Aug 29 2019
Aug 28 2019
Aug 27 2019
In D21412#466484, @kib wrote:Peter, could you, please, start testing this ?
Aug 25 2019
I ran the full stress2 test with D20814.61081.diff on amd64. LGTM.
I ran all of the stress2 tests with D21357.61095.diff on amd64. LGTM.
Aug 23 2019
I should be able to free up some test hardware later today.
Aug 21 2019
I ran a brief 2 hour test with D21320.61033.diff. No problems seen.
Aug 16 2019
D13671.60851.diff LGTM.
Aug 3 2019
Aug 1 2019
I ran tests for 15 hours on D20814.60344.diff without seeing any problems.
I completed a full stress2 test on amd64 without seeing any problems.
Jul 31 2019
With D20814.60322.diff I got:
20190731 19:27:23 all (1/1): mmap10.sh panic: vm_map_unwire: lookup failed cpuid = 12 time = 1564594061 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c96c5790 vpanic() at vpanic+0x19d/frame 0xfffffe00c96c57e0 panic() at panic+0x43/frame 0xfffffe00c96c5840 vm_map_wire_locked() at vm_map_wire_locked+0x69d/frame 0xfffffe00c96c5900 vm_map_wire() at vm_map_wire+0x40/frame 0xfffffe00c96c5930 kern_mlock() at kern_mlock+0x179/frame 0xfffffe00c96c5980 amd64_syscall() at amd64_syscall+0x2d6/frame 0xfffffe00c96c5ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00c96c5ab0
D20814.60315.diff
20190731 16:18:08 all (17/73): mmap10.sh panic: Lock vm map (user) not exclusively locked @ ../../../vm/vm_map.c:3102 cpuid = 12 time = 1564582693 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c7c80800 vpanic() at vpanic+0x19d/frame 0xfffffe00c7c80850 panic() at panic+0x43/frame 0xfffffe00c7c808b0 _sx_xunlock() at _sx_xunlock+0xab/frame 0xfffffe00c7c808d0 _vm_map_unlock() at _vm_map_unlock+0x75/frame 0xfffffe00c7c80900 vm_map_wire() at vm_map_wire+0x58/frame 0xfffffe00c7c80930 kern_mlock() at kern_mlock+0x179/frame 0xfffffe00c7c80980 amd64_syscall() at amd64_syscall+0x2d6/frame 0xfffffe00c7c80ab0
Jul 30 2019
I ran all of the threaded tests from stress2 on amd64 with D21088.60283.diff.
No problems seen.
Just a follow up. I'm currently testing your patch on amd64 where I have not yet seen any problems.
I got this after a 5 hour test on i386, followed by a shutdown:
panic: vm_fault_hold: fault on nofault entry, addr: 0x83326000 cpuid = 0 time = 1564486041 KDB: stack backtrace: db_trace_self_wrapper(e6666d,1bdcda4,0,23a17534,bd2eb1,...) at db_trace_self_wrapper+0x2a/frame 0x23a17508 kdb_backtrace(e,0,0,2b06100,2db5000,...) at kdb_backtrace+0x2e/frame 0x23a17568 vpanic(163709e,23a175b0,23a175b0,23a17680,12c2044,...) at vpanic+0x121/frame 0x23a17590 panic(163709e,158dad9,83326000,4895800,7000004,...) at panic+0x14/frame 0x23a175a4 vm_fault_hold(2db5000,83326000,2,0,0) at vm_fault_hold+0x2474/frame 0x23a17680 vm_fault(2db5000,83326000,2,0) at vm_fault+0x4a/frame 0x23a176a8 trap_pfault(8332680c) at trap_pfault+0xf3/frame 0x23a176f4 trap(23a177bc,8,28,28,83df8640,...) at trap+0x3c0/frame 0x23a177b0 calltrap() at 0xffc0316d/frame 0x23a177b0 --- trap 0xc, eip = 0xf7f255, esp = 0x23a177fc, ebp = 0x23a17814 --- funsetownlst(27e5ea50) at funsetownlst+0xa5/frame 0x23a17814 pgdelete(27e5ea6c,23c00a80,1651842,36,23a178f8,...) at pgdelete+0x60/frame 0x23a17834 leavepgrp(298d46a8,0,1651842,396,44c,...) at leavepgrp+0xd5/frame 0x23a1784c proc_reap(23c00a80,298d46a8,23a17ba4,36,1edeb50,...) at proc_reap+0x714/frame 0x23a178f8 proc_to_reap(23c00a80,298d46a8,7,0,0,...) at proc_to_reap+0x718/frame 0x23a17998 kern_wait6(23c00a80,7,0,0,23a17ba4,...) at kern_wait6+0x37a/frame 0x23a17a78 kern_wait(23c00a80,ffffffff,23a17ba4,36,0,...) at kern_wait+0x119/frame 0x23a17b80 sys_wait4(23c00a80,23c00d0c) at sys_wait4+0x68/frame 0x23a17c08 syscall(23a17ce8,3b,3b,3b,0,...) at syscall+0x304/frame 0x23a17cdc Xint0x80_syscall() at 0xffc033b7/frame 0x23a17cdc
Ran tests for another 23 hours on D20863.60217.diff. No problems seen.
Jul 27 2019
I tested D20814.60178.diff for 13 hours on amd64 without seeing any problems.
Jul 25 2019
The i386 tests finished with one further problem and an incomplete report: https://people.freebsd.org/~pho/stress/log/alc011.txt
Jul 24 2019
I finished the full stress2 tests on amd64 without any issues.
Jul 22 2019
Sure, happy to.
I ran tests on a GENERICish kernel for 12 hours without any issue, however a kernel build with "options KTR" does not compile:
Jul 21 2019
I still see the panic with D20863.59987.diff:
I ran a brief 4 hour test with this patch.
I completed a full stress2 test with D20947.59944.diff on amd64. No problems seen.
With D20863.59975.diff I got:
Jul 18 2019
A have completed a full stress2 test of D20833.59786.diff without seeing any problems.
Jul 17 2019
I have completed a full stress2 test with D20863.59745.diff on amd64 without seeing any problems.
Jul 15 2019
Some KTR output for a short test of D20863.59745.diff:
https://people.freebsd.org/~pho/stress/log/dougm051.txt
I tested D20949.59742.diff with all of the threaded tests I have, on i386. No problems seen.
Jul 14 2019
With D20863.59728.diff I got:
Jul 13 2019
With D20863.59658.diff I got:
Jul 12 2019
I finished the full stress2 test of D20893.59605.diff without finding any problems.
Jul 11 2019
The test has now run for 12 hours without any problems. I'll let the test run for the full ~48 hours.
Jul 10 2019
As far as I can tell the "panic: freeing free block" was introduced by r349777 and this patch didn't fix that.
I have not observed any other problems while testing with D20893.59572.diff
I have tested D20893.59572.diff on amd64 and briefly on i386.
I still see the "panic: freeing free block" on both a MEMGUARD and a GENERICish build.
Jul 9 2019
Looks like this patch is responsible for the panics I see.
20190709 11:56:55 all (1/1): sort.sh swap_pager: out of swap space swp_pager_getswapspace(32): failed Jul 9 12:03:41 mercat1 kernel: pid 3581 (sort), jid 0, uid 0, was killed: out of swap space Jul 9 12:03:42 mercat1 kernel: pid 3583 (sort), jid 0, uid 0, was killed: out of swap space panic: freeing free block: ffffc0, size 16, mask 1 cpuid = 7 time = 1562666624 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe102d95b270 vpanic() at vpanic+0x19d/frame 0xfffffe102d95b2c0 panic() at panic+0x43/frame 0xfffffe102d95b320 blst_meta_free() at blst_meta_free+0x12b/frame 0xfffffe102d95b360 blst_meta_free() at blst_meta_free+0x102/frame 0xfffffe102d95b3a0 blst_meta_free() at blst_meta_free+0x102/frame 0xfffffe102d95b3e0 blst_meta_free() at blst_meta_free+0x102/frame 0xfffffe102d95b420 blst_meta_free() at blst_meta_free+0x102/frame 0xfffffe102d95b460 blist_free() at blist_free+0x2e/frame 0xfffffe102d95b480 swp_pager_freeswapspace() at swp_pager_freeswapspace+0x8a/frame 0xfffffe102d95b4a0 swp_pager_meta_free_all() at swp_pager_meta_free_all+0xbb/frame 0xfffffe102d95b4f0 swap_pager_dealloc() at swap_pager_dealloc+0x115/frame 0xfffffe102d95b510 vm_object_terminate() at vm_object_terminate+0x27b/frame 0xfffffe102d95b560 vm_object_deallocate() at vm_object_deallocate+0x412/frame 0xfffffe102d95b5c0 vm_map_process_deferred() at vm_map_process_deferred+0x7f/frame 0xfffffe102d95b5e0 vm_map_remove() at vm_map_remove+0xc6/frame 0xfffffe102d95b610 vmspace_exit() at vmspace_exit+0xd3/frame 0xfffffe102d95b650 exit1() at exit1+0x5ad/frame 0xfffffe102d95b6c0 sigexit() at sigexit+0xdaf/frame 0xfffffe102d95b9a0 postsig() at postsig+0x336/frame 0xfffffe102d95ba70 ast() at ast+0x4c7/frame 0xfffffe102d95bab0 doreti_ast() at doreti_ast+0x1f/frame 0x7fffffffded0 KDB: enter: panic [ thread pid 3581 tid 100238 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db> x/s version version: FreeBSD 13.0-CURRENT #5 r349777: Tue Jul 9 11:37:16 CEST 2019\012 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/MEMGUARD\012 db>
The same test works fine on r349776.
In D20772#452692, @kib wrote:Thank you, Mark.
Peter, could you, please, run the tests one more time, hopefully this is the last.
I'll try again and avoid a panic, but in the meantime here's some trace: https://people.freebsd.org/~pho/stress/log/dougm047.txt
Jul 8 2019
So, this is unrelated to D20833.
In D20833#452429, @dougm wrote:The obvious questions are:
Does this happen without the patch in place?
Does this happen before r349777?
Came across this panic:
Jul 7 2019
I ran tests for 5 hours on i386 with D20833.59496.diff. No problems seen.
I have started a brief test run on i386 with D20833.59496.diff.
Jul 6 2019
I tested D20833.59460.diff for 5 hours without seeing any problems.
Jul 5 2019
I have run tests on D20579.59404.diff (with the typo fixed) for 8 hours. This included a buildworld / installworld.
cc -c -O2 -pipe -fno-strict-aliasing -g -nostdinc -I. -I../../.. -I../../../contrib/ck/include -I../../../contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD -MF.depend.subr_blist.o -MTsubr_blist.o -fdebug-prefix-map=./machine=/usr/src/sys/amd64/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member -mno-aes -mno-avx -std=iso9899:1999 -Werror ../../../kern/subr_blist.c ../../../kern/subr_blist.c:659:17: error: use of undeclared identifier 'BLIST_MAP_RADIX' if (maxcount % BLIST_MAP_RADIX != 0) ^ 1 error generated. *** Error code 1
Ran a brief test (5 hours) on D20833.59430.diff. No problems seen.
I have run tests on D20635.59379.diff for 24 hours without seeing any problems.
Jul 4 2019
In D20635#451659, @alc wrote:Peter, can you please retest this patch? I think that all of the open issues have been addressed.
Jul 3 2019
I ran tests on D20833.59317.diff for 14 hours. LGTM.
Jul 2 2019
Yes, fetch doesn't work well. Try using wget, which recovers from closed connections.
Fatal trap 12: page fault while in kernel mode cpuid = 0; current process = 4000 (tmlock) interrupt enabled, fault virtual address = 0x20 stack pointer = 0x28:0xfffffe00ad822860 resume, IOPL = 0 current process = 4002 (tmlock) code segment = base 0x0, limit 0xfffff, type 0x1b code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 trap number = 12 panic: page fault cpuid = 11 time = 1562057287 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00ac311520 vpanic() at vpanic+0x19d/frame 0xfffffe00ac311570 panic() at panic+0x43/frame 0xfffffe00ac3115d0 trap_fatal() at trap_fatal+0x39c/frame 0xfffffe00ac311630 trap_pfault() at trap_pfault+0x62/frame 0xfffffe00ac311680 trap() at trap+0x2b4/frame 0xfffffe00ac311790 calltrap() at calltrap+0x8/frame 0xfffffe00ac311790 --- trap 0xc, rip = 0xffffffff80f0db15, rsp = 0xfffffe00ac311860, rbp = 0xfffffe00ac311910 --- vm_map_wire_locked() at vm_map_wire_locked+0x155/frame 0xfffffe00ac311910 vm_map_wire() at vm_map_wire+0x40/frame 0xfffffe00ac311940 kern_mlock() at kern_mlock+0x179/frame 0xfffffe00ac311990 amd64_syscall() at amd64_syscall+0x291/frame 0xfffffe00ac311ab0
I have completed the selected swap tests and continued to run random tests for a total of 13H30.
No problems seen.
Jul 1 2019
This version got me past the test that kept triggering problems. I'll run some more tests ...
20190701 18:36:41 all (5/10): swap5.sh panic: swp_pager_force_pagein: Too many pages: 32 cpuid = 8 time = 1561999005 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00adc594c0 vpanic() at vpanic+0x19d/frame 0xfffffe00adc59510 panic() at panic+0x43/frame 0xfffffe00adc59570 swp_pager_force_pagein() at swp_pager_force_pagein+0x7f/frame 0xfffffe00adc596f0 swap_pager_swapoff_object() at swap_pager_swapoff_object+0xfe/frame 0xfffffe00adc59750 swap_pager_swapoff() at swap_pager_swapoff+0xf9/frame 0xfffffe00adc597d0 swapoff_one() at swapoff_one+0x15e/frame 0xfffffe00adc59820 sys_swapoff() at sys_swapoff+0x1d6/frame 0xfffffe00adc59990 amd64_syscall() at amd64_syscall+0x291/frame 0xfffffe00adc59ab0
I'll try Diff 59264.
D20635.59261.diff did not seem to fix the problem.
20190701 17:39:16 all (5/10): swap5.sh
With D20635.59244.diff I now see a deadlock:
https://people.freebsd.org/~pho/stress/log/dougm044.txt
20190701 07:18:49 all (2/70): mmap10.sh panic: _vm_map_clip_start: invalid clip of entry 0xfffff801496d34d0 cpuid = 9 time = 1561958339 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00ad7307a0 vpanic() at vpanic+0x19d/frame 0xfffffe00ad7307f0 panic() at panic+0x43/frame 0xfffffe00ad730850 _vm_map_clip_start() at _vm_map_clip_start+0x81/frame 0xfffffe00ad730890 vm_map_unwire() at vm_map_unwire+0x350/frame 0xfffffe00ad730960 sys_munlockall() at sys_munlockall+0x71/frame 0xfffffe00ad730990 amd64_syscall() at amd64_syscall+0x291/frame 0xfffffe00ad730ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00ad730ab0
Jun 30 2019
With D20635.59216.diff I see:
20190630 11:39:48 all (5/10): swap5.sh panic: swapoff: failed to locate 8036 swap blocks cpuid = 2 time = 1561887597 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00ad6c26a0 vpanic() at vpanic+0x19d/frame 0xfffffe00ad6c26f0 panic() at panic+0x43/frame 0xfffffe00ad6c2750 swap_pager_swapoff() at swap_pager_swapoff+0x194/frame 0xfffffe00ad6c27d0 swapoff_one() at swapoff_one+0x15e/frame 0xfffffe00ad6c2820 sys_swapoff() at sys_swapoff+0x1d6/frame 0xfffffe00ad6c2990 amd64_syscall() at amd64_syscall+0x291/frame 0xfffffe00ad6c2ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00ad6c2ab0 --- syscall (424, FreeBSD ELF64, sys_swapoff), rip = 0x8002f88ca, rsp = 0x7fffffffe4c8, rbp = 0x7fffffffe5f0 --- KDB: enter: panic [ thread pid 15529 tid 100290 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db> x/s version version: FreeBSD 13.0-CURRENT #1 r349552M: Sun Jun 30 10:48:06 CEST 2019\012 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 db>
The swap5.sh test scenario is a "Test with out of swap space" test using a 4g memory disk as swap.
On i386 I ran a buildworld / installworld + random stress2 tests for a total of 10H30.
On amd64 I ran random stress2 test for 10H30.
No problems seen.