Page MenuHomeFreeBSD

stat: Add option to list holes
ClosedPublic

Authored by des on Wed, Sep 10, 11:33 PM.
Tags
None
Referenced Files
F129578611: D52481.id161927.diff
Mon, Sep 22, 4:00 AM
Unknown Object (File)
Sun, Sep 21, 10:49 AM
Unknown Object (File)
Sun, Sep 21, 10:48 AM
Unknown Object (File)
Sun, Sep 21, 10:04 AM
Unknown Object (File)
Sun, Sep 21, 8:42 AM
Unknown Object (File)
Sun, Sep 21, 8:25 AM
Unknown Object (File)
Sun, Sep 21, 8:15 AM
Unknown Object (File)
Sun, Sep 21, 6:44 AM

Details

Summary

Add a new -h option that causes stat to print a list of holes for each
file argument.

Sponsored by: Klara, Inc.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 66970
Build 63853: arc lint + arc unit

Event Timeline

des requested review of this revision.Wed, Sep 10, 11:33 PM

This is a cool feature. Maybe it should be mentioned in lseek.2? That's a bit out-of-place, but it'd make this flag more discoverable.

usr.bin/stat/stat.1
145

I'd suggest referring to the lseek man page here: probably most readers won't know what the "virtual hole" is.

usr.bin/stat/stat.c
294

Should we check for unrelated flags being set, e.g., -f?

445–446

This line doesn't need to be wrapped.

kevans added inline comments.
usr.bin/stat/stat.c
1133

I think we should report the greater of l and sb.st_blksize -- ZFS reports a lower _PC_MIN_HOLE_SIZE for $reasons and you can have technically have a hole around that size for a file up until they reach the dataset recordsize, at which point you can't really have a hole less than the blocksize. Most filesystems, i would expect, have a PC_MIN_HOLE_SIZE >= sb.st_blksize and would generally not be affected if we do it.

usr.bin/stat/stat.c
1133

At least ZFS will have _PC_MIN_HOLE_SIZE <= sb.st_blksize. I'm not familiar with ZFS' implementation of holes, but in general a file's block size is dynamic; ZFS will try to use larger block for a file if doing so reduces fragmentation (i.e., the file is large) and space is available. _PC_MIN_HOLE_SIZE is just the smallest possible block size.

des marked 3 inline comments as done.

rf

des marked an inline comment as done.Thu, Sep 11, 2:39 PM
des added inline comments.
usr.bin/stat/stat.1
145

That's a direct quote from POSIX actually.

usr.bin/stat/stat.c
1133

That's not helpful as sb.st_blksize is also nonsense (roughly, the file size rounded up to the nearest multiple of 512) for files / directories smaller than the record size.

des marked an inline comment as done.Thu, Sep 11, 2:39 PM
markj added inline comments.
usr.bin/stat/stat.1
168

This applies to Fl l too I believe.

This revision is now accepted and ready to land.Thu, Sep 11, 4:37 PM
usr.bin/stat/stat.c
1133

I would argue that it would be useful there if I'm expanding a file and ZFS would only create holes aligned to that blocksize (rather than expanding the blocksize with the additional contents of the file). I don't recall off-hand how it behaves when operating on an existing file.

des marked 2 inline comments as done.Thu, Sep 11, 7:08 PM
des added inline comments.
usr.bin/stat/stat.c
1133

For files larger than the record size, st_blksize is the record size, which is also the hole granularity. But for files smaller than the record size, st_blksize appears to be the file size rounded up to the nearest multiple of 512. It is not necessarily a power of two or a divisor of the record size. It is a multiple of the reported minimum hole size (512), but you cannot reliably create holes in small files on ZFS. So no, it is not useful in any way.

des marked an inline comment as done.

rf

This revision now requires review to proceed.Thu, Sep 11, 7:09 PM
usr.bin/stat/stat.c
1133

It is a multiple of the reported minimum hole size (512), but you cannot reliably create holes in small files on ZFS

That in itself sounds like a solid reason for it to be useful. What are we using this information for, anyways? It's the smallest granularity of size and alignment for a hole, no? Two points:

  1. If I can't reliably create holes in small files anyways, why would I want to waste more time checking chunks of the smaller size if I'm exceedingly likely to not find anything of interest?
  2. As the file expands, though, same question: the granularity of hole becomes incredibly coarse in comparison and suddenly I'm spending more cycles trying to preserve or create holes that can't possibly exist.

There's of course other use-cases for this information, but this doesn't feel very useful for the kinds of things I need to do and I'll still have to maintain my own bits to get the information I really need anyways.

des marked an inline comment as done.Fri, Sep 12, 1:17 PM
des added inline comments.
usr.bin/stat/stat.c
1133

Kyle, I get the feeling that we're talking past each other.

To summarize, your proposal is to report the largest of fpathconf(fd, _PC_MIN_HOLE_SIZE) and sb.st_blksize as the hole boundary granularity. What I'm trying to tell you is that on ZFS, the former is always 512, and for files smaller than the record size, the latter is the current size of the file rounded up to a multiple of 512, which _in no way_ reflects the actual minimum hole size or hole boundary granularity.

If you still believe that “the greater of 512 or the file size rounded up to a multiple of 512” is useful information than I will ask you to please describe a use case for it.

I'll add that according to pathconf(3), ZFS should return 1, not 512, for _PC_MIN_HOLE_SIZE. We can deal with that separately.

des marked an inline comment as done.

special case for empty files

tweak comment and stop trying to process non-files

emaste added inline comments.
usr.bin/stat/stat.1
159

Would "inconsistent" be better? I imagine we're going to have consistency issues and report holes that have just been filled or exclude holes that have just been created, but that's short of "nonsense" I'd say.

des marked an inline comment as done.Fri, Sep 12, 2:16 PM
des added inline comments.
usr.bin/stat/stat.1
159

You can literally get a partial result followed by an error if the file gets truncated while we're reading it. That's more than just inconsistent. I suppose I can change that, though.

des marked an inline comment as done.

Try to recover gracefully in case of truncation. As a side effect, this removes the need for special-casing empty files.

This revision is now accepted and ready to land.Mon, Sep 15, 9:47 PM
This revision was automatically updated to reflect the committed changes.