makefs: Make it possible to silence warnings about duplicate paths

[markj]

Summary

When generating a VM image from an installworld mtree manifest, makefs
spits out several thousand warnings about duplicate paths in the
manifest. These are harmless and I don't see a good way to fix the
underlying duplication, so let's at least have a way to make makefs
quieter.

[brooks]

Seems fine.

IMO all of these are bugs in installworld and non were present when I first implemented manifest support, but there seems to be no will to do anything about them.

[emaste]

I came to the same conclusion (that it's really not feasible to "fix" installworld) for the pkgbase check script in 2a3bd0870869819354b5ea65761d13e5e2987697

[markj]

In D43513#992086, @brooks wrote:

Seems fine.

IMO all of these are bugs in installworld and non were present when I first implemented manifest support, but there seems to be no will to do anything about them.

I presumed that these warnings are basically just an artifact of the manifest implementation. I've been using that feature for a long time and don't remember a time where I didn't get errors like that.

I have no idea how to fix this properly. How would you go about it?

[imp]

In D43513#992112, @markj wrote:

In D43513#992086, @brooks wrote:

Seems fine.

IMO all of these are bugs in installworld and non were present when I first implemented manifest support, but there seems to be no will to do anything about them.

I presumed that these warnings are basically just an artifact of the manifest implementation. I've been using that feature for a long time and don't remember a time where I didn't get errors like that.

I have no idea how to fix this properly. How would you go about it?

When I tried for a project I was working on, I write a script that pre-processed the metalog to get rid of them.

Suppressing the warnings here is likely better since we can potentially warn when they are created in conflicting ways (which we likely should fix).

Getting it right in installworld seemed like the right thing to do, but perpetually tricky due to too many hacks and odd-edge-case quirks that our build system has, not all of which are obvious or clearly still relevant. Plus the difficulty at tracking down the duplicates and coming up with some sane way to detect it in real-time so the errors can't persist long (or aren't committed in the first place).

tl;dr: silencing the warnings here seems the next best thing...

[emaste]

AIUI the original intent is that all directories were to be created up front via etc/mtree and there would be no duplication. But after the addition of specific dir handling rules (e.g. a9e8c5c4b08ca673aef48cc10b45abc70c1033e0) it's no longer the case.

potentially warn when they are created in conflicting ways (which we likely should fix).

If METALOG entries have conflicting metadata it's an error and must be fixed. tools/pkgbase/metalog_reader.lua checks for and errors on this case. Duplicate identical directories don't even produce a warning (after the commit referenced earlier) for the same reason as Mark's proposed change here.

Plus the difficulty at tracking down the duplicates and coming up with some sane way to detect it in real-time so the errors can't persist long (or aren't committed in the first place).

Detecting the duplicates is trivial, it's fixing them that is the tricky part.

[brooks]

In D43513#992116, @emaste wrote:

AIUI the original intent is that all directories were to be created up front via etc/mtree and there would be no duplication. But after the addition of specific dir handling rules (e.g. a9e8c5c4b08ca673aef48cc10b45abc70c1033e0) it's no longer the case.

Right, I'd forgotten about this. This does beg the question: should the system mtree entries exist at all? In the pkg world I'm not sure they make sense except for specific empty directories and presumably those should be part of a pkg?

potentially warn when they are created in conflicting ways (which we likely should fix).

If METALOG entries have conflicting metadata it's an error and must be fixed. tools/pkgbase/metalog_reader.lua checks for and errors on this case. Duplicate identical directories don't even produce a warning (after the commit referenced earlier) for the same reason as Mark's proposed change here.

Conflicting metadata is certainly a bug as is any duplicate file entry.

Plus the difficulty at tracking down the duplicates and coming up with some sane way to detect it in real-time so the errors can't persist long (or aren't committed in the first place).

Detecting the duplicates is trivial, it's fixing them that is the tricky part.

Currently they are hard to track down. I wonder if we want to add a new (always ignored) mtree field that that we could stuff .CURDIR in to make this more debuggable?

The grand vision I'll certainly never have time for would be:

1. Import libmtree (https://github.com/mratajsky/libmtree) and use it everywhere so we have a single mtree parser/emitter
2. Add a third, more query friendly mtree format (probably sqlite based)
3. Use the binary format to detect true conflicts at insert time so we always fail stop