Page MenuHomeFreeBSD

rm(1): Formalize non-functional status of -P flag
ClosedPublic

Authored by cem on Nov 8 2018, 9:22 AM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, Apr 25, 9:48 AM
Unknown Object (File)
Mon, Apr 8, 8:29 AM
Unknown Object (File)
Thu, Mar 28, 11:03 AM
Unknown Object (File)
Jan 29 2024, 1:06 PM
Unknown Object (File)
Jan 10 2024, 2:39 PM
Unknown Object (File)
Jan 7 2024, 9:20 PM
Unknown Object (File)
Dec 22 2023, 10:48 PM
Unknown Object (File)
Dec 10 2023, 8:52 PM

Details

Summary

-P was introduced in 4.4BSD-Lite2 around 1994. It overwrote file contents
with a pass of 0xff, 0x00, then 0xff, in a low effort attempt to "really
delete" files.

It has no user-visible effect; at the end of the day, the file is unlinked via
the filesystem. Furthermore, the utility of overwriting files with patterned
data is extremely limited due to caveats at every layer of the stack[0] and
therefore mostly futile. At the least, three passes is likely wasteful on
modern hardware[1]. It could also be seen as a violation of the "Unix
Philosophy" to do one thing per tiny, composable program.

Since 1994, FreeBSD has left it alone; OpenBSD replaced it with a single
pass of arc4random(3) output in 2012[2]; and NetBSD implemented partial, but
explicitly incomplete support for U.S. DoD 5220.22-M, "National Industrial
Security Program Operating Manual" in 2004[3].

NetBSD's enhanced comment above rm_overwrite makes a strong case for removing
the flag entirely:

This is an expensive way to keep people from recovering files from your
non-snapshotted FFS filesystems using fsdb(8). Really. No more.

It is impossible to actually conform to the exact procedure given in
[NISPOM] if one is overwriting a file, not an entire disk, because the
procedure requires examination and comparison of the disk's defect lists.
Any program that claims to securely erase *files* while conforming to the
standard, then, is not correct.

Furthermore, the presence of track caches, disk and controller write
caches, and so forth make it extremely difficult to ensure that data have
actually been written to the disk, particularly when one tries to repeatedly
overwrite the same sectors in quick succession. We call fsync(), but
controllers with nonvolatile cache, as well as IDE disks that just plain lie
about the stable storage of data, will defeat this.

[NISPOM] requires physical media destruction, rather than any technique of
the sort attempted here, for secret data.

As a first step towards evental removal, make it a placebo. It's not like
it was serving any security function with more code.

If you are security conscious and need to erase your disks, use a woodchipper.

If you are cheap and your secrets aren't worth spending any money on, use a
single iteration of gshred (part of GNU coreutils) (or just dd if=/dev/zero).

[0]: https://www.ru.nl/publish/pages/909282/draft-paper.pdf
[1]: https://commons.erau.edu/cgi/viewcontent.cgi?article=1131&context=jdfsl
[2]: https://github.com/openbsd/src/commit/7c5c57ba81b5fe8ff2d4899ff643af18c
[3]: https://github.com/NetBSD/src/commit/fdf0a7a25e59af958fca1e2159921562cd

Test Plan

No problem with POSIX -- only -ifR are required.

Diff Detail

Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 20713
Build 20124: arc lint + arc unit

Event Timeline

LGTM although I wonder about adding a warning if it's used.

This revision is now accepted and ready to land.Nov 8 2018, 9:30 AM
markj added inline comments.
bin/rm/rm.1
237

I'd put this under COMPATIBILITY.

bin/rm/rm.c
130

It'd probably be worth printing a warning to stderr if this option is specified.

cem marked 2 inline comments as done.Nov 8 2018, 6:57 PM
cem added inline comments.
bin/rm/rm.1
237

Oh sure, that seems reasonable. Will do.

bin/rm/rm.c
130

(@Daniel as well)

Hm, my concern with printing any additional warning is that it is a functional (observable) change in behavior as far as scripting is concerned. Would something like a syslog message be a good compromise, or do you think that's a worse idea?

Is there prior art in this space? I know some commands just ignore obsolete options and don't print anything. Do we have a good example of the warning case? Thanks.

cem marked an inline comment as done.

Move compatibility note to COMPATIBILITY section, per markj

This revision now requires review to proceed.Nov 8 2018, 11:49 PM
cem marked an inline comment as done.Nov 8 2018, 11:57 PM
cem added inline comments.
bin/rm/rm.c
130

Some examples:

xz(1): --fast/--best "provided only for backwards compatibility. Avoid using these options." No warning printed when used.

look(1): "The -a and ... flags are ignored for compatibility." No warning is printed when used.

as(1): "-D Ignored. This option is accepted for ... compatibility." No warning is printed.

diff(1): "--speed-large-files: Stub option for compatibility with GNU diff". No warning is printed.

I'm leaning towards no warning. Are there some good counter examples? Maybe UPDATING / relnotes: yes?

markj added inline comments.
bin/rm/rm.c
130

Fair enough. I tried to think of some examples and couldn't come up with any. You're right that it'd probably break scripts.

This revision is now accepted and ready to land.Nov 9 2018, 12:05 AM
This revision was automatically updated to reflect the committed changes.

This is a massive POLA violation. What the hell. You took a flag that was intended to make <some effort> to overwrite data and made it SILENTLY DO NOTHING. That's insane.

  • Not all files are written to more than once; the file may have been written once and overwriting its blocks may actually be sufficient.
  • Not all filesystems are ZFS
  • Not all users have root to wipe a disk
  • Not all users want or need to WIPE AN ENTIRE DISK to wipe 1 file
  • Not all files warrant a full cleanse; a half hearted effort is still better than nothing.
  • Not all users have the quota to fill the disk with dd
  • Not all filesystems have snapshots
  • You mentioned what OpenBSD and NetBSD did (to keep it) but ended with basically "fuck it let's remove it".

Yes the multi-pass was pointless. But 1 pass isn't entirely pointless. In the end this flag was _better than nothing_ and now it's a silent lie.

This flag was not advertised as some DoD or NISPOM standard-abiding feature. The note about it not being a security feature was added in 2010 with explanations on caveats.

While FreeBSD is about backwards-compatibility and POLA, this silent change may actually expose data for people where unexpected. That failure case is worth actually making this PRINT SOMETHING or ERROR with an invalid flag. Or better yet just be left alone. As I mentioned the case of a once-written file would be perfectly fine with this feature no? Ok sure there's the cache issues but that's more limited in its capacity but NOW THERES THE MAIN DATA BLOCKS ISSUES which is far greater in its exposure capacity.

Yes the multi-pass was pointless. But 1 pass isn't entirely pointless. In the end this flag was _better than nothing_ and now it's a silent lie.

I would also like to inquire if this has been revisited. I see that Bryan's comment was asked months after the PR was merged, but I agree with the entirety of his post. Was more detail uncovered as to why not even a single overwrite was still left in place for rm -P? Please advise. I am new in this community, but I know there are still quite a few people since this changes was made who are still using (and advocating others to use) the -P flag in public forums.

A single overwrite would still be of great use, for those who are wanting the common case:

  1. on a mac (yes, I do love FreeBSD, but mac users are by far the largest volume of users affected by this)
  2. non-encrypted filesystem (not sure if this even matters: https://www.fsl.cs.stonybrook.edu/docs/secdel/secdel.html)
  3. wanting to remove a file, but knowing a simple unlink will leave that file's disk blocks containing the file's contents
  4. the file's removal is done purely out of best-efforts prudence. It is acceptable if some filesystem cache / artifacts may have some of the file's contents for a short period. Within a few weeks or months of normal system usage, the file's contents very likely will be actually purged.

This is best-effort is commonly the case for all kinds of things in my industry, where PHI is the sensitive information we are purging. In these cases it is best-effort that is the key, as it is fully understood that there are time windows in which this sensitive data will persist. As long as best-efforts are taken to "shred" that sensitive data, then you are compliant.

My concern is that the trend has been first the removal of srm because there existed rm -P, now rm -P has been scuttled, leaving no easily-found mechanism to do at least the minimum protection against recovery of unlinked files.

What is the best approach to escalate this? I can issue a PR that is simply a reversion of this, but I would like to follow the proper course of discussion used by this community. Post this on Conpherence?