makefs_zfs regression tests are all using the same zpool name, so are not compliant with parallel execution.
Details
Without this patch:
VM# echo "parallelism=8" >> /etc/kyua/kyua.conf VM# cd /usr/tests/usr.sbin/makefs/ VM# kyua test makefs_zfs_tests makefs_zfs_tests:empty_dir -> failed: atf-check failed; see the output of the test for details [0.290s] makefs_zfs_tests:dataset_removal -> failed: atf-check failed; see the output of the test for details [0.322s] makefs_zfs_tests:hard_links -> failed: atf-check failed; see the output of the test for details [0.322s] makefs_zfs_tests:file_sizes -> failed: atf-check failed; see the output of the test for details [0.492s] makefs_zfs_tests:empty_fs -> passed [0.417s] makefs_zfs_tests:multi_dataset_1 -> failed: atf-check failed; see the output of the test for details [0.323s] makefs_zfs_tests:basic -> failed: atf-check failed; see the output of the test for details [0.783s] makefs_zfs_tests:autoexpand -> failed: atf-check failed; see the output of the test for details [0.838s] makefs_zfs_tests:multi_dataset_2 -> passed [0.408s] makefs_zfs_tests:multi_dataset_3 -> failed: atf-check failed; see the output of the test for details [0.285s] makefs_zfs_tests:multi_dataset_4 -> failed: atf-check failed; see the output of the test for details [0.222s] makefs_zfs_tests:multi_staging_2 -> failed: atf-check failed; see the output of the test for details [0.289s] makefs_zfs_tests:snapshot -> failed: atf-check failed; see the output of the test for details [0.195s] makefs_zfs_tests:multi_staging_1 -> passed [0.385s] makefs_zfs_tests:soft_links -> failed: atf-check failed; see the output of the test for details [0.252s] makefs_zfs_tests:root_props -> failed: atf-check failed; see the output of the test for details [0.806s] makefs_zfs_tests:long_file_name -> passed [1.371s] makefs_zfs_tests:indirect_dnode_array -> passed [3.035s] makefs_zfs_tests:reproducible -> passed [3.499s] Results file id is usr_tests_usr.sbin_makefs.20220824-230817-731224 Results saved to /root/.kyua/store/results.usr_tests_usr.sbin_makefs.20220824-230817-731224.db 6/19 passed (13 failed)
Diff Detail
- Lint
Lint Skipped - Unit
Tests Skipped
Event Timeline
makefs_zfs regression tests are all using the same zpool name
The poolname contains the PID, so I guess kyua is executing multiple tests in the same process context? How can I see the full test stdout/stderr when running multiple tests in parallel?
Yes indeed, the PID are all diferrents so the poolname. The problem is elsewhere:
# kyua report --verbose ===> Execution context Current directory: /usr/tests/usr.sbin/makefs Environment variables: BLOCKSIZE=K EDITOR=vi GROUP=wheel HOME=/root HOST=VM HOSTTYPE=FreeBSD LANG=C.UTF-8 LOGNAME=root MACHTYPE=x86_64 MAIL=/var/mail/root MM_CHARSET=UTF-8 OSTYPE=FreeBSD PAGER=less PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin PWD=/usr/tests/usr.sbin/makefs SHELL=/bin/csh SHLVL=1 TERM=vt100 USER=root VENDOR=amd ===> makefs_zfs_tests:autoexpand Result: failed: atf-check failed; see the output of the test for details Start time: 2022-08-25T00:19:46.948214Z End time: 2022-08-25T00:19:47.796297Z Duration: 0.848s Metadata: allowed_architectures is empty allowed_platforms is empty description is empty has_cleanup = true is_exclusive = false required_configs is empty required_disk_space = 0 required_files is empty required_memory = 0 required_programs is empty required_user is empty timeout = 300 Standard output: Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/2/work/mnt ] Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/2/work/inputs ] Executing command [ mkdir -m 0755 -p a/b/1 ] Executing command [ ln -s a/b c ] Executing command [ touch d ] Executing command [ ln d e ] Executing command [ touch .f ] Executing command [ mkdir .g ] Executing command [ dd if=/dev/zero of=i count=1000 bs=1 ] Executing command [ touch klmn ] Executing command [ touch opqr ] Executing command [ touch stuv ] Executing command [ install -m 0755 /dev/null wxyz ] Executing command [ touch 0b00000001 ] Executing command [ touch 0b00000010 ] Executing command [ touch 0b00000011 ] Executing command [ touch 0b00000100 ] Executing command [ touch 0b00000101 ] Executing command [ touch 0b00000110 ] Executing command [ touch 0b00000111 ] Executing command [ touch 0b00001000 ] Executing command [ touch 0b00001001 ] Executing command [ touch 0b00001010 ] Executing command [ touch 0b00001011 ] Executing command [ touch 0b00001100 ] Executing command [ touch 0b00001101 ] Executing command [ touch 0b00001110 ] Executing command [ dd if=/dev/zero of=1.file bs=1 count=1 oseek=1 conv=sparse ] Executing command [ dd if=/dev/zero of=512.file bs=1 count=1 oseek=512 conv=sparse ] Executing command [ dd if=/dev/zero of=2048.file bs=1 count=1 oseek=2048 conv=sparse ] Executing command [ dd if=/dev/zero of=10240.file bs=1 count=1 oseek=10240 conv=sparse ] Executing command [ dd if=/dev/zero of=524288.file bs=1 count=1 oseek=524288 conv=sparse ] Executing command [ makefs -t zfs -o nowarn=true -s 10737418240 -o mssize=134217728 -o rootpath=/ -o poolname=makefstest30970 /tmp/kyua.78oQpz/2/work/test.img /tmp/kyua.78oQpz/2/work/inputs ] Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/2/work/test.img ] Executing command [ zpool import -R /tmp/kyua.78oQpz/2/work/mnt makefstest30970 ] Standard error: Fail: incorrect exit status: 1, expected: 0 stdout: stderr: cannot import 'makefstest30970': no such pool available Files left in work directory after failure: inputs, md.output, mnt, test.img cat: /tmp/kyua.78oQpz/2/work/poolname: No such file or directory cannot open '': name must begin with a letter ===> makefs_zfs_tests:basic Result: failed: atf-check failed; see the output of the test for details Start time: 2022-08-25T00:19:46.949610Z End time: 2022-08-25T00:19:47.786611Z Duration: 0.837s Metadata: allowed_architectures is empty allowed_platforms is empty description is empty has_cleanup = true is_exclusive = false required_configs is empty required_disk_space = 0 required_files is empty required_memory = 0 required_programs is empty required_user is empty timeout = 300 Standard output: Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/3/work/mnt ] Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/3/work/inputs ] Executing command [ mkdir -m 0755 -p a/b/1 ] Executing command [ ln -s a/b c ] Executing command [ touch d ] Executing command [ ln d e ] Executing command [ touch .f ] Executing command [ mkdir .g ] Executing command [ dd if=/dev/zero of=i count=1000 bs=1 ] Executing command [ touch klmn ] Executing command [ touch opqr ] Executing command [ touch stuv ] Executing command [ install -m 0755 /dev/null wxyz ] Executing command [ touch 0b00000001 ] Executing command [ touch 0b00000010 ] Executing command [ touch 0b00000011 ] Executing command [ touch 0b00000100 ] Executing command [ touch 0b00000101 ] Executing command [ touch 0b00000110 ] Executing command [ touch 0b00000111 ] Executing command [ touch 0b00001000 ] Executing command [ touch 0b00001001 ] Executing command [ touch 0b00001010 ] Executing command [ touch 0b00001011 ] Executing command [ touch 0b00001100 ] Executing command [ touch 0b00001101 ] Executing command [ touch 0b00001110 ] Executing command [ dd if=/dev/zero of=1.file bs=1 count=1 oseek=1 conv=sparse ] Executing command [ dd if=/dev/zero of=512.file bs=1 count=1 oseek=512 conv=sparse ] Executing command [ dd if=/dev/zero of=2048.file bs=1 count=1 oseek=2048 conv=sparse ] Executing command [ dd if=/dev/zero of=10240.file bs=1 count=1 oseek=10240 conv=sparse ] Executing command [ dd if=/dev/zero of=524288.file bs=1 count=1 oseek=524288 conv=sparse ] Executing command [ makefs -t zfs -o nowarn=true -s 10g -o rootpath=/ -o poolname=makefstest30971 /tmp/kyua.78oQpz/3/work/test.img /tmp/kyua .78oQpz/3/work/inputs ] Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/3/work/test.img ] Executing command [ zpool import -R /tmp/kyua.78oQpz/3/work/mnt makefstest30971 ] Standard error: Fail: incorrect exit status: 1, expected: 0 stdout: stderr: cannot import 'makefstest30971': no such pool available Files left in work directory after failure: inputs, md.output, mnt, test.img cat: /tmp/kyua.78oQpz/3/work/poolname: No such file or directory cannot open '': name must begin with a letter ===> makefs_zfs_tests:empty_dir Result: failed: atf-check failed; see the output of the test for details Start time: 2022-08-25T00:19:46.957465Z End time: 2022-08-25T00:19:47.283094Z Duration: 0.326s Metadata: allowed_architectures is empty allowed_platforms is empty description is empty has_cleanup = true is_exclusive = false required_configs is empty required_disk_space = 0 required_files is empty required_memory = 0 required_programs is empty required_user is empty timeout = 300 Standard output: Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/5/work/mnt ] Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/5/work/inputs ] Executing command [ makefs -t zfs -o nowarn=true -s 10g -o rootpath=/ -o poolname=makefstest30975 /tmp/kyua.78oQpz/5/work/test.img /tmp/kyua .78oQpz/5/work/inputs ] Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/5/work/test.img ] Executing command [ zpool import -R /tmp/kyua.78oQpz/5/work/mnt makefstest30975 ] Standard error: Fail: incorrect exit status: 1, expected: 0 stdout: stderr: cannot import 'makefstest30975': pool already exists Files left in work directory after failure: inputs, md.output, mnt, test.img cat: /tmp/kyua.78oQpz/5/work/poolname: No such file or directory cannot open '': name must begin with a letter ===> makefs_zfs_tests:empty_fs Result: failed: atf-check failed; see the output of the test for details Start time: 2022-08-25T00:19:46.965942Z End time: 2022-08-25T00:19:47.298847Z Duration: 0.333s Metadata: allowed_architectures is empty allowed_platforms is empty description is empty has_cleanup = true is_exclusive = false required_configs is empty required_disk_space = 0 required_files is empty required_memory = 0 required_programs is empty required_user is empty timeout = 300 Standard output: Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/6/work/mnt ] Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/6/work/inputs ] Executing command [ makefs -t zfs -o nowarn=true -s 10g -o rootpath=/ -o poolname=makefstest30987 /tmp/kyua.78oQpz/6/work/test.img /tmp/kyua .78oQpz/6/work/inputs ] Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/6/work/test.img ] Executing command [ zpool import -R /tmp/kyua.78oQpz/6/work/mnt makefstest30987 ] Standard error: Fail: incorrect exit status: 1, expected: 0 stdout: stderr: cannot import 'makefstest30987': no such pool available Files left in work directory after failure: inputs, md.output, mnt, test.img cat: /tmp/kyua.78oQpz/6/work/poolname: No such file or directory cannot open '': name must begin with a letter ===> makefs_zfs_tests:file_sizes Result: failed: atf-check failed; see the output of the test for details Start time: 2022-08-25T00:19:46.972400Z End time: 2022-08-25T00:19:47.543610Z Duration: 0.571s Metadata: allowed_architectures is empty allowed_platforms is empty description is empty has_cleanup = true is_exclusive = false required_configs is empty required_disk_space = 0 required_files is empty required_memory = 0 required_programs is empty required_user is empty timeout = 300 Standard output: Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/7/work/mnt ] Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/7/work/inputs ] Executing command [ makefs -t zfs -o nowarn=true -s 10g -o rootpath=/ -o poolname=makefstest30998 /tmp/kyua.78oQpz/7/work/test.img /tmp/kyua .78oQpz/7/work/inputs ] Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/7/work/test.img ] Executing command [ zpool import -R /tmp/kyua.78oQpz/7/work/mnt makefstest30998 ] Standard error: Fail: incorrect exit status: 1, expected: 0 stdout: stderr: cannot import 'makefstest30998': no such pool available Files left in work directory after failure: inputs, md.output, mnt, test.img cat: /tmp/kyua.78oQpz/7/work/poolname: No such file or directory cannot open '': name must begin with a letter
I see now. The problem is that pools created by makefs always have the same GUID, so they can't be imported simultaneously. I added a parameter to makefs to specify a PRNG seed, and if I modify the tests to use it, they can run in parallel without problems. So, I don't think it'll be necessary to serialize the makefs_zfs tests.
I tried several variants of this idea but was always able to trigger occasional collisions by running the ZFS tests in a loop overnight. So I ended up just committing a variant of this patch, together with some unrelated bug fixes.