Diff Detail
- Repository
- rG FreeBSD src repository
- Lint
Lint Skipped - Unit
Tests Skipped - Build Status
Buildable 71406 Build 68289: arc lint + arc unit
Event Timeline
The idea is to have an external setup/teardown files which could be implemented much simpler compared to the test itself. For example, setup is done with usual shell and tools from the base, while the test itself could be written in C.
It looks for a setup file named the same way as the test program but with .setup suffix, it removes the need to configure it explicitly. If a specific test case within the same program needs special setup then newly added setup metadata can be used to override the suffix just for that test case (the included demo shows this). If setup metadata does not look like a dot-starting suffix then it's treated as a file path relative to the test program location.
This is a working prototype with the first demo so that we can experiment and discuss expectations and needs. For the beginning the patch is very minimal -- it works only for execenv=jail and supports setup files only (no teardown support).
The demo test cases:
root@taarch64:~ # cd /usr/tests/sys/netpfil/pf/ root@taarch64:/usr/tests/sys/netpfil/pf # kldload if_epair if_bridge root@taarch64:/usr/tests/sys/netpfil/pf # kyua debug demo:one Executing command [ ping -c1 192.0.2.239 ] setup file KYUA_TESTDIR: /usr/tests/sys/netpfil/pf setup file PWD: /tmp/kyua.WnQ6Qp/2/work setup file args: demo:one -> passed root@taarch64:/usr/tests/sys/netpfil/pf # kyua debug demo:two Executing command [ ping -c1 192.0.2.1 ] setup file KYUA_TESTDIR: /usr/tests/sys/netpfil/pf setup file PWD: /tmp/kyua.r8lVVV/2/work setup file args: 192.0.2.1 demo:two -> passed root@taarch64:/usr/tests/sys/netpfil/pf # kyua debug demo:three Executing command [ ping -c1 192.0.2.100 ] demo:three -> passed
There is an additional idea to think of in the future to add a new metadata named like execenv_jail_attachto=<jail_name> or so. The goal is, if an external setup configures a complex set of jails and the test is expected to be running within a specific child jail only, then such metadata would ask Kyua to do so.
Thanks a lot for working on that! I miss the feature a lot.
Sorry if a dumb question. Trying to understand the Makefile logic. The statement
${PACKAGE}SCRIPTS+= demo.setup \
demo.atfsh_like_setupdoes it apply to all tests in the Makefile or there is some magic that makes the first word before the dot to limit application only to a test that is named as this word? Where does this logic live?
Perhaps, I would better outline the reasoning behind the patch, it would answer the question and help the involved parties to think of the eventual UI/UX we would like to have:
- Okay, the idea is to have setup/teardown process which is not part of a test program and is provided in another file; how would Kyua locate it?
- It seems to be a good idea to make Kyua simply look for a file within the same directory as the test program and named the same way but with .setup added to the name
- So that both a test and its setup code are listed together, e.g.:
/usr/tests/sys/newfeat/subsystem1$ ls first_test first_test.setup second_test second_test.setup
- I've quickly checked the source code -- we do not have files named like *.setup in tests/
- Such "by convention" approach is expected to be better than explicit definition "by configuration" in Makefile per test or so. And it's good to have a standard naming in the test suite.
- At the same time I had questions like what if a specific test program/case needs to refer a setup file with a non-usual name, or, let's say, a group of tests need the same setup file. For that we have a new metadata named "setup".
- The "setup" metadata string is treated as a file path relative to the test program's directory
- e.g. atf_set setup 'prepare_test_env.sh' is used by test_timeout test program:
/usr/tests/sys/newfeat/subsystem2$ ls prepare_test_env.sh test_timeout
- and another example where we define on the Kyuafile level a common setup script for all tests in the directory by adding TEST_METADATA+= setup="../common/prepare_jails.sh" to the Makefile:
/usr/tests/sys/newfeat/subsystem3$ ls test_this test_that verify_a_corner_case /usr/tests/sys/newfeat/subsystem3$ ls ../common prepare_jails.sh
- Also, the "setup" metadata has a trick to check if the string begins with a dot and is not followed by another dot or slash then it's treated as a suffix, not like a full name. For example, if an ATF test program named a_nice_test uses atf_set setup '.fixture' then Kyua would look for a setup file named a_nice_test.fixture located in the same directory. That is, just renaming both the test program and the corresponding setup file should work without setup file location re-configuration.
- We might decide to remove this trick, but currently it serves as the way to define .setup as a suffix by default for all tests, as we do not know test names beforehand. It feels to be better this way than hardcoding this default with a special logic in Kyua.
- Later I was thinking what if a group of test programs refer to the same common setup binary/script but each test wants to run it with a specific argument, e.g. to prepare test env specific per-test way. For that "setup" metadata can accept arguments separated with a white-space from the file path/suffix.
- e.g. atf_set setup '../common/prepare_jails.sh 192.0.2.2'
- All the same applies to the teardown, but it will be implemented after we decide on the final UI/UX design
Yep, it's a lot, but it tries to describe current design as much as possible, because I would like others to challenge it in order to make sure the feature fits the actual production needs.
Looks like the logic covers all imaginable cases. Of course as result it is a bit complex. But if properly documented should be fine.
Waiting for review from the test people!