User Details
- User Since
- Aug 6 2014, 5:32 AM (583 w, 4 d)
Sat, Oct 4
Sun, Sep 21
Tue, Sep 16
Mon, Sep 15
Sun, Sep 14
Aug 1 2025
Jul 16 2025
Jun 29 2025
I ran tests with D51085.id157729.diff for 18 hours without seeing any issues.
Jun 1 2025
May 26 2025
I ran fsck tests with D50520.156022.patch added, without observing any issues.
May 18 2025
I have been running tests with D50392.155617.patch for 13 hours without seeing any issues
May 16 2025
May 12 2025
May 7 2025
Looks good to me.
Apr 29 2025
I ran tests with D47114.154425.patch for 5 hours without seeing any issues.
Apr 25 2025
Apr 24 2025
I ran the syzkaller reproducer for an hour with D49987.154201.patch added. I added a further 4 hours of random test cases.
I did not observe any issues.
Apr 23 2025
20250423 09:20:24 all (1/1): syzkaller75.sh panic: vm_pager_assert_in: page 0xfffffe0000b5fcc0 is mapped cpuid = 11 time = 1745393009 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe016df1d7b0 vpanic() at vpanic+0x136/frame 0xfffffe016df1d8e0 panic() at panic+0x43/frame 0xfffffe016df1d940 vm_pager_assert_in() at vm_pager_assert_in+0x1fa/frame 0xfffffe016df1d980 vm_pager_get_pages() at vm_pager_get_pages+0x3d/frame 0xfffffe016df1d9d0 vm_fault() at vm_fault+0x745/frame 0xfffffe016df1db40 vm_map_wire_locked() at vm_map_wire_locked+0x385/frame 0xfffffe016df1dbf0 vm_mmap_object() at vm_mmap_object+0x2fd/frame 0xfffffe016df1dc50 vn_mmap() at vn_mmap+0x152/frame 0xfffffe016df1dce0 kern_mmap() at kern_mmap+0x621/frame 0xfffffe016df1ddc0 sys_mmap() at sys_mmap+0x42/frame 0xfffffe016df1de00 amd64_syscall() at amd64_syscall+0x15a/frame 0xfffffe016df1df30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe016df1df30 --- syscall (0, FreeBSD ELF64, syscall), rip = 0x823efb7fa, rsp = 0x821f52f68, rbp = 0x821f52f90 --- KDB: enter: panic [ thread pid 37673 tid 108519 ] Stopped at kdb_enter+0x33: movq $0,0x104d7a2(%rip) db> x/s version version: FreeBSD 15.0-CURRENT #0 main-n276680-d14036ea424d-dirty: Wed Apr 23 09:10:41 CEST 2025 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO db> Uptime: 8m32s
Doug,
I have mailed you the test scenario. I will start a test on the latest version of HEAD.
Apr 22 2025
The syzkaller test still triggers a panic:
Apr 19 2025
I got this panic, which seems unrelated to me, after 8 hours of testing:
Apr 3 2025
Feb 27 2025
I completed a full stress2 test run with D49103.151425.patch without seeing any issues.
Feb 24 2025
I got this with D49103.151389.patch:
20250224 11:38:23 all (254/953): mmap8.sh panic: vm_reserv_from_object: msucc doesn't succeed pindex cpuid = 4 time = 1740393505 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01087809b0 vpanic() at vpanic+0x136/frame 0xfffffe0108780ae0 panic() at panic+0x43/frame 0xfffffe0108780b40 vm_reserv_from_object() at vm_reserv_from_object+0xc9/frame 0xfffffe0108780b50 vm_reserv_alloc_page() at vm_reserv_alloc_page+0x72/frame 0xfffffe0108780bb0 vm_page_alloc_domain_after() at vm_page_alloc_domain_after+0x140/frame 0xfffffe0108780c40 vm_page_alloc_after() at vm_page_alloc_after+0x54/frame 0xfffffe0108780cb0 vm_fault_copy_entry() at vm_fault_copy_entry+0x32f/frame 0xfffffe0108780d60 vm_map_protect() at vm_map_protect+0x72b/frame 0xfffffe0108780df0 sys_mprotect() at sys_mprotect+0x9f/frame 0xfffffe0108780e00 amd64_syscall() at amd64_syscall+0x15a/frame 0xfffffe0108780f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108780f30 --- syscall (74, FreeBSD ELF64, mprotect), rip = 0x822c0ed7a, rsp = 0x820d0ee78, rbp = 0x820d0eea0 --- KDB: enter: panic [ thread pid 82131 tid 102074 ] Stopped at kdb_enter+0x33: movq $0,0x104ed52(%rip) db>
It's easy to reproduce on my test box.
Feb 10 2025
I ran tests with D48588.150722.patch for two days without observing any problems.
Feb 4 2025
I ran all of the stress2 tests using nullfs and also ronald@'s test scenario. LGTM.
Jan 29 2025
FYI: I came across this i386 problem after commit 0078df5f0258.
Trying to mount root from ufs:/dev/vtbd0p1 [rw]... WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. WARNING: 32-bit kernels are deprecated and may be removed in FreeBSD 15.0. panic: vm_page_t 0x5d49be0 phys_addr mismatch ffffffffb8779000 00000000b8779405 cpuid = 3 time = 1738177579 KDB: stack backtrace: db_trace_self_wrapper(7,114beb40,5d49be0,ffffffff,ffffffff,...) at db_trace_self_wrapper+0x28/frame 0x113f6744 vpanic(148badd,113f6780,113f6780,113f67f0,13a54e9,...) at vpanic+0xf4/frame 0x113f6760 panic(148badd,5d49be0,b8779000,ffffffff,b8779405,...) at panic+0x14/frame 0x113f6774 pmap_pae_remove_pages(114128e8) at pmap_pae_remove_pages+0x599/frame 0x113f67f0 exec_new_vmspace(113f6994,188247c) at exec_new_vmspace+0x1cb/frame 0x113f6810 exec_elf32_imgact(113f6994,16d4801,113f6984,0,0,...) at exec_elf32_imgact+0x5e4/frame 0x113f6874 kern_execve(114beb40,113f6a8c,0,1141284c) at kern_execve+0x72c/frame 0x113f6a74 sys_execve(114beb40,114bedf0,114beb40,13ae913,114bede4,...) at sys_execve+0x4e/frame 0x113f6ac8 syscall(113f6ba8,3b,3b,3b,402aaf,...) at syscall+0x1e6/frame 0x113f6b9c Xint0x80_syscall() at 0xffc03479/frame 0x113f6b9c --- syscall (59, FreeBSD ELF32, execve), eip = 0x47e53f, esp = 0xffbfe848, ebp = 0xffbfe858 --- KDB: enter: panic [ thread pid 17 tid 100080 ] Stopped at kdb_enter+0x34: movl $0,kdb_why db> x/s version version: FreeBSD 15.0-CURRENT #0 main-n275068-0078df5f0258-dirty: Wed Jan 29 18:44:45 CET 2025\012 pho@mercat1.netperf.freebsd.org:/mnt25/obj/usr/src/i386.i386/sys/PHO\012 db>
Jan 27 2025
With this patch I caught this:
I ran tests. for a day with D45409.149981.patch. LGTM.
Dec 4 2024
Dec 3 2024
I'm puzzled by how your change could work for you.
I too have been thinking about removing the "Missing EOF hole" comment, but stalled because I still am not sure that the output is correct.
Isn't there supposed to be a virtual hole at the end of a file? See for example https://docs.oracle.com/cd/E86824_01/html/E54765/lseek-2.html
That does not work for me. This is what I get on a pristine install:
Nov 28 2024
Nov 20 2024
I completed a full stress2 test without seing any issues.
Nov 18 2024
Nov 14 2024
I do net see any panics with D47559.146396.patch
Nov 13 2024
I ran all of the stress2 SU+J tests without seeing any issues with this patch.
Oct 22 2024
Oct 20 2024
Oct 18 2024
Oops. Forgot the panic string :)
I cannot boot with D47150.145065.patch:
Oct 17 2024
I ran tests with D47150.144979.patch for 17 hours before getting this seemingly unrelated ext2fs panic:
https://people.freebsd.org/~pho/stress/log/log0555.txt
Oct 14 2024
I ran tests with D46963.144796.patch added for 6 hours without seeing any issues.
Oct 12 2024
I ran tests with D46963.144614.patch added for 14 hours. I did not observe any issues.
Oct 11 2024
I ran the test swapoff4.sh for 3 hours with your patch and didn't see any issues.
Not sure if the panic is related to this patch?
20241011 13:49:02 all (664/970): swapoff4.sh Oct 11 13:50:22 mercat1 kernel: pid 90548 (swap), jid 0, uid 0, was killed: failed to reclaim memory Oct 11 13:50:23 mercat1 kernel: pid 90555 (swap), jid 0, uid 0, was killed: failed to reclaim memory Oct 11 13:50:25 mercat1 kernel: pid 90550 (swap), jid 0, uid 0, was killed: failed to reclaim memory panic: Assertion (object->flags & OBJ_SWAP) != 0 failed at ../../../vm/swap_pager.c:564 cpuid = 5 time = 1728647425 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01d8174990 vpanic() at vpanic+0x13f/frame 0xfffffe01d8174ac0 panic() at panic+0x43/frame 0xfffffe01d8174b20 swapoff_one() at swapoff_one+0x8a9/frame 0xfffffe01d8174d00 kern_swapoff() at kern_swapoff+0x1ab/frame 0xfffffe01d8174e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01d8174f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01d8174f30 --- syscall (582, FreeBSD ELF64, swapoff), rip = 0xdc6777123ba, rsp = 0xdc6748c87e8, rbp = 0xdc6748c8920 ---
Oct 3 2024
Sep 16 2024
Sep 13 2024
I ran a full stress2 test with D45627.143204.patch added and saw no (new) issues.
Aug 30 2024
Aug 29 2024
Aug 23 2024
I ran a 14 hour test with D45627.142138.patch without finding any issues.
Aug 17 2024
Aug 16 2024
Aug 15 2024
Ran a 8 hour test with D45627.142095.patch. No problems seen.
Aug 14 2024
I ran a brief test with D45987.142064.patch without seeing any problems.
Aug 8 2024
cc -c -O2 -pipe -fno-strict-aliasing -g -nostdinc -I. -I../../.. -I../../../contrib/ck/include -I../../../contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD -MF.depend.subr_rangeset.o -MTsubr_rangeset.o -fdebug-prefix-map=./machine=/usr/src/sys/amd64/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -fdebug-prefix-map=./i386=/usr/src/sys/i386/include -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-4 -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wswitch -Wno-error=tautological-compare -Wno-error=empty-body -Wno-error=parentheses-equality -Wno-error=unused-function -Wno-error=pointer-sign -Wno-error=shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length -mno-aes -mno-avx -std=gnu99 -Werror ../../../kern/subr_rangeset.c ../../../kern/subr_rangeset.c:329:25: error: use of undeclared identifier 'src_rs' 329 | pctrie_iter_init(&it, &src_rs->rs_trie); | ^ 1 error generated. *** Error code 1
I ran tests with D45987.141850.patch without seeing any issues.
Aug 5 2024
Here's a panic with D45627.141818.patch:
Jul 30 2024
I ran tests with D45627.141545.patch for 10 hours, without seeing any problems.
Jul 25 2024
I ran stress tests for 10 hours with D46099.141332.patch. I did not observe any issues.
Jul 20 2024
Jul 19 2024
Jul 18 2024
I ran tests with D45987.141024.patch. I ran all of the tmpfs test scenarios in a loop for 15 hours, without seeing any problems.
Jul 17 2024
I ran tests with D45627.141008.patch for a day, without seeing any problems.
Jul 16 2024
Here's a panic in subr_pctrie.c:100
20240716 09:00:01 all (1/958): pfl4.sh Jul 16 09:02:51 mercat1 kernel: pid 12456 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page Jul 16 09:04:09 mercat1 kernel: pid 16534 (rw), uid 2007 inumber 53830 on /mnt12: filesystem full Kernel page fault with the following non-sleepable locks held: exclusive rw vmobject (vmobject) r = 0 (0xfffff80013bce738) locked @ vm/vm_object.c:1333 stack backtrace: #0 0xffffffff80bc82bc at witness_debugger+0x6c #1 0xffffffff80bc94b3 at witness_warn+0x403 #2 0xffffffff81076ff0 at trap_pfault+0x80 #3 0xffffffff81049258 at calltrap+0x8 #4 0xffffffff80ba6655 at pctrie_remove+0x1e5 #5 0xffffffff80ba7715 at pctrie_iter_remove+0x145 #6 0xffffffff80eeca49 at SWAP_PCTRIE_ITER_REMOVE+0x19 #7 0xffffffff80ee8520 at swp_pager_meta_free+0x380 #8 0xffffffff80ee7e3a at swap_pager_freespace_pgo+0x7a #9 0xffffffff80f10f19 at vm_object_madvise+0x149 #10 0xffffffff80f05e5a at vm_map_madvise+0x3ea #11 0xffffffff81077918 at amd64_syscall+0x158 #12 0xffffffff81049b6b at fast_syscall_common+0xf8
I was able to reproduce a similar "out of pages" watchdog panic with a pristine kernel and a new test scenario.
So, it seems to me that the watchdog issue is unrelated to your D45627.140892.patch?
Here's the one with a pristine kernel: https://people.freebsd.org/~pho/stress/log/log0540.txt
Jul 15 2024
This is how I ran this test:
While running tests with D45627.140892.patch I got this: "The watchdog fired with a one hour timeout"
https://people.freebsd.org/~pho/stress/log/log0539.txt
Jul 9 2024
jemalloc(3) is failing again:
20240709 10:24:07 all: crossmp4.sh <jemalloc>: jemalloc_base.c:190: Failed assertion: "extent_bsize_get(extent) >= *gap_size + size" <jemalloc>: jemalloc_base.c:190: Failed assertion: "extent_bsize_get(extent) >= *gap_size + size" <jemalloc>: /usr/src/contrib/jemalloc/include/jemalloc/internal/tcache_inlines.h:52: Failed assertion: "tcache_success == (ret != NULL)" <jemalloc>: /usr/src/contrib/jemalloc/include/jemalloc/internal/tcache_inlines.h:52: Failed assertion: "tcache_success == (ret != NULL)"
The watchdog, with a timeout of one hour, fired: https://people.freebsd.org/~pho/stress/log/log0538.txt
I'll resume testing ...
Jul 6 2024
I ran tests with D45398.140613.patch for 9 hours without seeing any problems.
Jul 5 2024
In my experience, it is not uncommon for the test environment to have a big impact on finding errors.
I got this panic with union19.sh:
20240705 20:25:38 all (1/1): unionfs19.sh VNASSERT failed: !__builtin_expect(((_Generic(*(&(dvp)->v_irflag), short: (*(volatile u_short *)(&(dvp)->v_irflag)), u_short: VNASSERT failed: !__builtin_expect(((_Generic(*(&(dvp)->v_irflag), short: (*(volatile u_short *)(&(dvp)->v_irflag)), u_short: (*(volatile u_short *)(&(dvp)->v_irflag))) & 0x0001) != 0), 0) not true at ../../../kern/vfs_cache.c:2481 (cache_enter_time) VNASSERT failed: !__builtin_expect(((_Generic(*(&(dvp)->v_irflag), short: (*(volatile u_short *)(&(dvp)->v_irflag)), u_short: VNASSERT failed: !__builtin_expect(((_Generic(*(&(dvp)->v_irflag), short: (*(volatile u_short *)(&(dvp)->v_irflag)), u_short: (*(volatile u_short *)(&(dvp)->v_irflag))) & 0x0001) != 0), 0) not true at ../../../kern/vfs_cache.c:2481 (cache_enter_time) VNASSERT failed: !__builtin_expect(((_Generic(*(&(dvp)->v_irflag), short: (*(volatile u_short *)(&(dvp)->v_irflag)), u_short: (*(volatile u_short *)(&(dvp)->v_irflag))) & 0x0001) != 0), 0) not true at ../../../kern/vfs_cache.c:2481 (cache_enter_time) 0xfffffe016f682bb8: (*(volatile u_short *)(&(dvp)->v_irflag))) & 0x0001) != 0), 0) not true at ../../../kern/vfs_cache.c:2481 (cache_enter_time) (*(volatile u_short *)(&(dvp)->v_irflag))) & 0x0001) != 0), 0) not true at ../../../kern/vfs_cache.c:2481 (cache_enter_time) 0xfffffe016f6d04b0: 0xfffffe016f1aa068: type VBAD state VSTATE_DEAD op 0xffffffff818ac760 usecount 2, writecount 0, refcount 1 seqc users 10xfffffe016ffba068: type VBAD state VSTATE_DEAD op 0xffffffff818ac760
I ran a mix of 48 tests with D45781.140542.patch for 13 hours. I saw no problems with this.
Jul 4 2024
These are my observations with running the stress2 swapoff.sh test in a loop on real hardware:
Jul 3 2024
With D45781.140474.patch added I got a deadlock after 6h30:
https://people.freebsd.org/~pho/stress/log/log0534.txt
It's not clear to me if this is related to your patch, so I'll repeat the test with a pristine kernel.
Jul 2 2024
This is the setup I use for stress testing on both real HW and bhyve (4 CPUs and 6GB RAM):
By capping RAM to 8 GB I was able to get a "hang". Unfortunately this is AFAIK a know issue when using memory disks: