It would also be nice if pjdfstest would fail if there is a new optional feature not listed in the configuration. Currently, if someone adds a new feature to pjdfstest, these tests won't enable it unless someone remembers to come along and update the configurations.

As I mentioned by email, some tests need more than one unprivileged user. Some need up to three. We can't really eliminate that need, but we can change the configured users.

Instead of putting all of these test programs under tests/sys/fs/pjdfstest , would it be possible to put them under separate directories for each file system? Like tests/sys/fs/tmpfs/pjdfstest.sh ? That would simplify running regression tests for a particular file system, for developers who aren't familiar with pjdfstest.

In D56605#1296058, @asomers wrote:

Instead of putting all of these test programs under tests/sys/fs/pjdfstest , would it be possible to put them under separate directories for each file system? Like tests/sys/fs/tmpfs/pjdfstest.sh ? That would simplify running regression tests for a particular file system, for developers who aren't familiar with pjdfstest.

Yeah, I could do that... I'm skeptical though that there can ever be one place that contains all regression tests you might want to run for a particular filesystem. How do you run all zfs tests for instance? There are some tests in sbin/bectl and usr.sbin/makefs that have found ZFS bugs. My stance on this problem is that in general you should just run the entire test suite, and we should focus on making that relatively fast and easy to do. To make it fast I've been trying to make tests run in parallel whenever possible, and to make it easy(ier) I wrote bricoler (https://github.com/markjdb/bricoler).

In D56605#1296083, @markj wrote:

In D56605#1296058, @asomers wrote:

Instead of putting all of these test programs under tests/sys/fs/pjdfstest , would it be possible to put them under separate directories for each file system? Like tests/sys/fs/tmpfs/pjdfstest.sh ? That would simplify running regression tests for a particular file system, for developers who aren't familiar with pjdfstest.

Yeah, I could do that... I'm skeptical though that there can ever be one place that contains all regression tests you might want to run for a particular filesystem. How do you run all zfs tests for instance? There are some tests in sbin/bectl and usr.sbin/makefs that have found ZFS bugs. My stance on this problem is that in general you should just run the entire test suite, and we should focus on making that relatively fast and easy to do. To make it fast I've been trying to make tests run in parallel whenever possible, and to make it easy(ier) I wrote bricoler (https://github.com/markjdb/bricoler).

Ok, we can do it this way if you prefer, then.

In D56605#1296048, @asomers wrote:

As I mentioned by email, some tests need more than one unprivileged user. Some need up to three. We can't really eliminate that need, but we can change the configured users.

Hmm, so we have "nobody" and "tests"... I guess we could just use another system user, maybe "daemon"? Alternately, the port could create a pjdfstest user, or we could require the test runner to configure one somehow (i.e., skip the tests unless a pjdfstest user exists).

In D56605#1296174, @asomers wrote:

In D56605#1296143, @markj wrote:

In D56605#1296048, @asomers wrote:

As I mentioned by email, some tests need more than one unprivileged user. Some need up to three. We can't really eliminate that need, but we can change the configured users.

Hmm, so we have "nobody" and "tests"... I guess we could just use another system user, maybe "daemon"? Alternately, the port could create a pjdfstest user, or we could require the test runner to configure one somehow (i.e., skip the tests unless a pjdfstest user exists).

I'm in favor of creating that user in the port.

Yeah, I think that makes the most sense.

One more related question: should the rewrite call itself pjdfstest, or something else? Is it going to replace the upstream implementation?

Yes, the plan is that this will replace the sh-based implementation. So I don't want to completely rename it. Or I suppose we could just retire the sh-based implementation, and call this one "pjdfstest2" or something.
Also, while this diff LGTM, you shouldn't commit it just yet, until we resolve all of the issues you opened on Github. That's because some of those might change the config file format.

In D56605#1297897, @asomers wrote:

Yes, the plan is that this will replace the sh-based implementation. So I don't want to completely rename it. Or I suppose we could just retire the sh-based implementation, and call this one "pjdfstest2" or something.

pjdfstest2 seems fine. I have no real preference, just wondering.

Also, while this diff LGTM, you shouldn't commit it just yet, until we resolve all of the issues you opened on Github. That's because some of those might change the config file format.

Sure, no rush. I wasn't going to commit this until pjdfstest is available as a binary package.

Please note that the new version requires _rust_. That's a pretty large dependency to require for CI, which doesn't exist already today.
This is part of the reason why I was more keen on doing python+pytest integration: python is more common in CI and pytest is on the docket for kyua integration (it's on my TODO list because I want the atf_python stuff to go away...).

In D56605#1298243, @ngie wrote:

Please note that the new version requires _rust_. That's a pretty large dependency to require for CI, which doesn't exist already today.
This is part of the reason why I was more keen on doing python+pytest integration: python is more common in CI and pytest is on the docket for kyua integration (it's on my TODO list because I want the atf_python stuff to go away...).

I don't follow: the tests require a compiled rust executable, which will be available via the ports tree. There is no dependency on rust itself.

@asomers I retried the tests with the pjdfstest built from ports. All of the tests pass except the UFS1 one:

root@freebsd:/tmp/kyua.uO4a2P/2/work # pjdfstest -c pjdfstest.toml -p mnt
utimensat::utime_now_write_perm                                               ok
utimensat::changes_timestamps::regular                                        ok
utimensat::changes_timestamps::socket                                         ok
utimensat::subsecond                                                          ok
utimensat::utime_now_nobody                                                   ok
utimensat::changes_timestamps::dir                                            ok
utimensat::nobody                                                             ok
utimensat::utime_now_root                                                     ok
utimensat::order                                                              ok
utimensat::follow_symlink                                                     ok
utimensat::root                                                               ok
Abort trap
root@freebsd:/tmp/kyua.uO4a2P/2/work # echo $?
134