Differential D38668
Further updates to UFS/FFS superblock integrity checks when reading a superblock Authored by mckusick on Feb 19 2023, 2:19 AM. Tags None Referenced Files
Details
Some additional filesystem fuzzing by Bob Prohaska has turned up ways to circumvent the existing filesystem checks. They can both crash the kernel and cause fsck_ffs(8) to segment fault when trying to fix them. These changes should resolve those problems without rejecting any valid filesystems. Check with submitter and have Peter Holm run his suite of filesystem checks.
Diff Detail
Event TimelineComment Actions Note that the three FS's that exhibit these problems are available at: Comment Actions This seems to have broken "-O1". For example: newfs -O1 -b 4096 -f 512 md10
** /dev/md10
** Last Mounted on /mnt
** Phase 1 - Check Blocks and Sizes
UFS1 cylinder group 0 failed: cgp->cg_iusedoff ("174") != (&cgp->cg_space[0] - (u_char *)(&cgp->cg_firstfield)) ("168")
UFS1 cylinder group 0 failed: cgp->cg_old_btotoff ("168") != cgp->cg_iusedoff ("174")
CYLINDER GROUP 0: INTEGRITY CHECK FAILED
REBUILD CYLINDER GROUP? yesComment Actions Fix problems found by Peter Holm. Fine tune and consolodate reported errors when fixing broken filesystems. Comment Actions During tests I found this problem with what appers to be a looping fsck_ffs: 11:22 /tmp $ uname -a FreeBSD mercat1.netperf.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n261120-49e3972afc111-dirty: Sat Feb 25 07:00:12 CET 2023 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO amd64 11:22 /tmp $ (cd /usr/src; git diff | grep +++) +++ b/sbin/fsck_ffs/dir.c +++ b/sbin/fsck_ffs/fsck.h +++ b/sbin/fsck_ffs/fsutil.c +++ b/sbin/fsck_ffs/inode.c +++ b/sbin/fsck_ffs/pass1.c +++ b/sbin/fsck_ffs/pass2.c +++ b/sbin/fsck_ffs/setup.c +++ b/sbin/fsck_ffs/suj.c +++ b/sys/ufs/ffs/ffs_subr.c 11:23 /tmp $ gunzip < fsck10.sh.diskimage.looping.20230225T093317.gz > /work/diskimage 11:23 /tmp $ fsck_ffs -fy /work/diskimage | head -50 /work/diskimage IS NOT A DISK DEVICE CONTINUE? yes UFS1 superblock failed: fs->fs_old_ncyl (1) != fs->fs_ncg (1073741825) UFS1 superblock failed: (u_int64_t)fs->fs_ipg * fs->fs_ncg (274877907200) > (((int64_t)(1)) << 32) - INOPB(fs) (4294967040) Attempted recovery for standard superblock: failed Attempt extraction of recovery data from standard superblock. Try cg 0 at sblock loc 64 UFS1 superblock failed: fs->fs_old_inodefmt (262146) != FS_44INODEFMT (2) Try cg 1 at sblock loc 1088 Try cg 2 at sblock loc 2112 Try cg 3 at sblock loc 3136 Try cg 4 at sblock loc 4160 Try cg 5 at sblock loc 5184 Try cg 6 at sblock loc 6208 Try cg 7 at sblock loc 7232 Try cg 8 at sblock loc 8256 Try cg 9 at sblock loc 9280 Try cg 10 at sblock loc 10304 Try cg 11 at sblock loc 11328 Try cg 12 at sblock loc 12352 Try cg 13 at sblock loc 13376 Try cg 14 at sblock loc 14400 Try cg 15 at sblock loc 15424 Try cg 16 at sblock loc 16448 Try cg 17 at sblock loc 17472 Try cg 18 at sblock loc 18496 Try cg 19 at sblock loc 19520 Try cg 20 at sblock loc 20544 Try cg 21 at sblock loc 21568 Try cg 22 at sblock loc 22592 Try cg 23 at sblock loc 23616 Try cg 24 at sblock loc 24640 Try cg 25 at sblock loc 25664 Try cg 26 at sblock loc 26688 Try cg 27 at sblock loc 27712 Try cg 28 at sblock loc 28736 Try cg 29 at sblock loc 29760 Try cg 30 at sblock loc 30784 Try cg 31 at sblock loc 31808 Try cg 32 at sblock loc 32832 Try cg 33 at sblock loc 33856 Try cg 34 at sblock loc 34880 Try cg 35 at sblock loc 35904 Try cg 36 at sblock loc 36928 Try cg 37 at sblock loc 37952 Try cg 38 at sblock loc 38976 Try cg 39 at sblock loc 40000 Try cg 40 at sblock loc 41024 11:24 /tmp $ https://people.freebsd.org/~pho/fsck10.sh.diskimage.looping.20230225T093317.gz Comment Actions Fix for Peter Holm's latest evil filesystem. When trying to check just the minimal set of fields to make a superblock usable I failed to verify fs_ncg which then made the attempt to read the summary information take approximately forever. Comment Actions Yes, thank you. That fixed the issue for me. I ran some more tests and uploaded some images you may find interesting: I also ran into a panic while testing, but unfortunately the disk image was lost. Comment Actions I found a disk image to reproduce this panic: Enter full pathname of shell or RETURN for /bin/sh: # sh /tmp/cleanclaim.sh + gunzip + fsck -fy /work/diskimage.3 /work/diskimage.3 is not a char device /work/diskimage.3 IS NOT A DISK DEVICE CONTINUE? yes ** /work/diskimage.3 ** Last Mounted on /mnt11 ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames I OUT OF RANGE out-of-range inode number 402 UNEXPECTED SOFT UPDATE INCONSISTENCY REMOVE? yes ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts LINK COUNT DIR I=146 OWNER=root MODE=40755 SIZE=512 MTIME=Feb 28 06:46 2023 COUNT 2 SHOULD BE 1 ADJUST? yes ** Phase 5 - Check Cyl groups 91 files, 91 used, 348 free (52 frags, 37 blocks, 11.8% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** + mdconfig -a -t vnode -f /work/diskimage.3 -u 10 + mkdir -p /mnt10 + mount /dev/md10 /mnt10 + sleep 1 + rm -rf /mnt10/1 /mnt10/10 /mntpanic: softdep_update_inodeblock inconsistent ip 0xfffffe01444adb30 i_nlink 1 inodedep 0xfffff8027abf9780 id_nlinkdelta 2 cpuid = 8 time = 1677573511 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0143809850 vpanic() at vpanic+0x152/frame 0xfffffe01438098a0 panic() at panic+0x43/frame 0xfffffe0143809900 softdep_update_inodeblock() at softdep_update_inodeblock+0x49a/frame 0xfffffe0143809950 ffs_update() at ffs_update+0x22d/frame 0xfffffe01438099b0 ffs_truncate() at ffs_truncate+0x724/frame 0xfffffe0143809b90 ufs_inactive() at ufs_inactive+0x1aa/frame 0xfffffe0143809bd0 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x60/frame 0xfffffe0143809c00 vinactivef() at vinactivef+0xda/frame 0xfffffe0143809c40 vput_final() at vput_final+0x2a7/frame 0xfffffe0143809c90 kern_frmdirat() at kern_frmdirat+0x2f1/frame 0xfffffe0143809e00 amd64_syscall() at amd64_syscall+0x15a/frame 0xfffffe0143809f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143809f--- syscall (137, FreeBSD ELF64, rmdir), rip = 0x3285da7aac1a, rsp = 0x3285d899cf38, rbp = 0x3285d899cf70 --- KDB: enter: panic [ thread pid 3113 tid 100244 ] Stopped at kdb_enter+0x32: movq $0,0x1290a13(%rip) db> x/s version version: FreeBSD 14.0-CURRENT #1 main-n261149-a4b92fefd2507-dirty: Mon Feb 27 09:49:36 CET 2023\012 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 db> https://people.freebsd.org/~pho/fsck11.sh.diskimage.20230228T064402.gz Comment Actions
It turns out that this existing patch fixes that crash. The bad inode that fsck finds is the inode number for the '.' entry in directory inode #146. Previously it removed the offending entry which left the directory with a link count of 1 which then tripped up the kernel logic that knows that directories should have a link count of at least 2. My fix notices when the bad inode number is '.' (or '..') and fixes it rather than removing the entry. So with this patch your disk image now cleans up correctly and works as it should. Note that one of your tests on a UFS2+SU+J filesystem with error "cglookup: out of range cylinder group 103564'' checks as clean but then when run again is not clean. Here the problem is that the first fsck run uses just the journal to clean up. The journal only cleans up expected errors and the error is due to some random field getting flipped. Thus (like all journaling systems) the error is not found by the journal fsck run. The second fsck run does a full scan of the filesystem and cleans up the smashed entry. Error cleanups of this sort are why it is necessary to run full fsck's periodically (or the equivalent `scrub' on ZFS). |