Page MenuHomeFreeBSD

Prevent makefs_zfs regression tests to run parrallel
AbandonedPublic

Authored by olivier on Aug 24 2022, 11:11 PM.
Tags
None
Referenced Files
Unknown Object (File)
Dec 20 2023, 5:27 AM
Unknown Object (File)
Aug 17 2023, 7:09 PM
Unknown Object (File)
May 29 2023, 7:49 AM
Unknown Object (File)
Mar 26 2023, 12:27 AM
Unknown Object (File)
Feb 17 2023, 4:20 PM
Unknown Object (File)
Dec 27 2022, 9:53 AM
Unknown Object (File)
Dec 15 2022, 7:43 PM
Subscribers
None

Details

Reviewers
markj
Group Reviewers
tests
Summary

makefs_zfs regression tests are all using the same zpool name, so are not compliant with parallel execution.

Test Plan

Without this patch:

VM# echo "parallelism=8" >> /etc/kyua/kyua.conf
VM# cd /usr/tests/usr.sbin/makefs/
VM# kyua test makefs_zfs_tests
makefs_zfs_tests:empty_dir  ->  failed: atf-check failed; see the output of the test for details  [0.290s]
makefs_zfs_tests:dataset_removal  ->  failed: atf-check failed; see the output of the test for details  [0.322s]
makefs_zfs_tests:hard_links  ->  failed: atf-check failed; see the output of the test for details  [0.322s]
makefs_zfs_tests:file_sizes  ->  failed: atf-check failed; see the output of the test for details  [0.492s]
makefs_zfs_tests:empty_fs  ->  passed  [0.417s]
makefs_zfs_tests:multi_dataset_1  ->  failed: atf-check failed; see the output of the test for details  [0.323s]
makefs_zfs_tests:basic  ->  failed: atf-check failed; see the output of the test for details  [0.783s]
makefs_zfs_tests:autoexpand  ->  failed: atf-check failed; see the output of the test for details  [0.838s]
makefs_zfs_tests:multi_dataset_2  ->  passed  [0.408s]
makefs_zfs_tests:multi_dataset_3  ->  failed: atf-check failed; see the output of the test for details  [0.285s]
makefs_zfs_tests:multi_dataset_4  ->  failed: atf-check failed; see the output of the test for details  [0.222s]
makefs_zfs_tests:multi_staging_2  ->  failed: atf-check failed; see the output of the test for details  [0.289s]
makefs_zfs_tests:snapshot  ->  failed: atf-check failed; see the output of the test for details  [0.195s]
makefs_zfs_tests:multi_staging_1  ->  passed  [0.385s]
makefs_zfs_tests:soft_links  ->  failed: atf-check failed; see the output of the test for details  [0.252s]
makefs_zfs_tests:root_props  ->  failed: atf-check failed; see the output of the test for details  [0.806s]
makefs_zfs_tests:long_file_name  ->  passed  [1.371s]
makefs_zfs_tests:indirect_dnode_array  ->  passed  [3.035s]
makefs_zfs_tests:reproducible  ->  passed  [3.499s]

Results file id is usr_tests_usr.sbin_makefs.20220824-230817-731224
Results saved to /root/.kyua/store/results.usr_tests_usr.sbin_makefs.20220824-230817-731224.db

6/19 passed (13 failed)

Diff Detail

Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

olivier created this revision.

makefs_zfs regression tests are all using the same zpool name

The poolname contains the PID, so I guess kyua is executing multiple tests in the same process context? How can I see the full test stdout/stderr when running multiple tests in parallel?

makefs_zfs regression tests are all using the same zpool name

The poolname contains the PID, so I guess kyua is executing multiple tests in the same process context? How can I see the full test stdout/stderr when running multiple tests in parallel?

Yes indeed, the PID are all diferrents so the poolname. The problem is elsewhere:

# kyua report --verbose
===> Execution context
Current directory: /usr/tests/usr.sbin/makefs
Environment variables:
    BLOCKSIZE=K
    EDITOR=vi
    GROUP=wheel
    HOME=/root
    HOST=VM
    HOSTTYPE=FreeBSD
    LANG=C.UTF-8
    LOGNAME=root
    MACHTYPE=x86_64
    MAIL=/var/mail/root
    MM_CHARSET=UTF-8
    OSTYPE=FreeBSD
    PAGER=less
    PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
    PWD=/usr/tests/usr.sbin/makefs
    SHELL=/bin/csh
    SHLVL=1
    TERM=vt100
    USER=root
    VENDOR=amd
===> makefs_zfs_tests:autoexpand
Result:     failed: atf-check failed; see the output of the test for details
Start time: 2022-08-25T00:19:46.948214Z
End time:   2022-08-25T00:19:47.796297Z
Duration:   0.848s

Metadata:
    allowed_architectures is empty
    allowed_platforms is empty
    description is empty
    has_cleanup = true
    is_exclusive = false
    required_configs is empty
    required_disk_space = 0
    required_files is empty
    required_memory = 0
    required_programs is empty
    required_user is empty
    timeout = 300

Standard output:
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/2/work/mnt ]
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/2/work/inputs ]
Executing command [ mkdir -m 0755 -p a/b/1 ]
Executing command [ ln -s a/b c ]
Executing command [ touch d ]
Executing command [ ln d e ]
Executing command [ touch .f ]
Executing command [ mkdir .g ]
Executing command [ dd if=/dev/zero of=i count=1000 bs=1 ]
Executing command [ touch klmn ]
Executing command [ touch opqr ]
Executing command [ touch stuv ]
Executing command [ install -m 0755 /dev/null wxyz ]
Executing command [ touch 0b00000001 ]
Executing command [ touch 0b00000010 ]
Executing command [ touch 0b00000011 ]
Executing command [ touch 0b00000100 ]
Executing command [ touch 0b00000101 ]
Executing command [ touch 0b00000110 ]
Executing command [ touch 0b00000111 ]
Executing command [ touch 0b00001000 ]
Executing command [ touch 0b00001001 ]
Executing command [ touch 0b00001010 ]
Executing command [ touch 0b00001011 ]
Executing command [ touch 0b00001100 ]
Executing command [ touch 0b00001101 ]
Executing command [ touch 0b00001110 ]
Executing command [ dd if=/dev/zero of=1.file bs=1 count=1 oseek=1 conv=sparse ]
Executing command [ dd if=/dev/zero of=512.file bs=1 count=1 oseek=512 conv=sparse ]
Executing command [ dd if=/dev/zero of=2048.file bs=1 count=1 oseek=2048 conv=sparse ]
Executing command [ dd if=/dev/zero of=10240.file bs=1 count=1 oseek=10240 conv=sparse ]
Executing command [ dd if=/dev/zero of=524288.file bs=1 count=1 oseek=524288 conv=sparse ]
Executing command [ makefs -t zfs -o nowarn=true -s 10737418240 -o mssize=134217728 -o rootpath=/ -o poolname=makefstest30970 /tmp/kyua.78oQpz/2/work/test.img /tmp/kyua.78oQpz/2/work/inputs ]
Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/2/work/test.img ]
Executing command [ zpool import -R /tmp/kyua.78oQpz/2/work/mnt makefstest30970 ]

Standard error:
Fail: incorrect exit status: 1, expected: 0
stdout:

stderr:
cannot import 'makefstest30970': no such pool available

Files left in work directory after failure: inputs, md.output, mnt, test.img
cat: /tmp/kyua.78oQpz/2/work/poolname: No such file or directory
cannot open '': name must begin with a letter
===> makefs_zfs_tests:basic
Result:     failed: atf-check failed; see the output of the test for details
Start time: 2022-08-25T00:19:46.949610Z
End time:   2022-08-25T00:19:47.786611Z
Duration:   0.837s

Metadata:
    allowed_architectures is empty
    allowed_platforms is empty
    description is empty
    has_cleanup = true
    is_exclusive = false
    required_configs is empty
    required_disk_space = 0
    required_files is empty
    required_memory = 0
    required_programs is empty
    required_user is empty
    timeout = 300

Standard output:
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/3/work/mnt ]
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/3/work/inputs ]
Executing command [ mkdir -m 0755 -p a/b/1 ]
Executing command [ ln -s a/b c ]
Executing command [ touch d ]
Executing command [ ln d e ]
Executing command [ touch .f ]
Executing command [ mkdir .g ]
Executing command [ dd if=/dev/zero of=i count=1000 bs=1 ]
Executing command [ touch klmn ]
Executing command [ touch opqr ]
Executing command [ touch stuv ]
Executing command [ install -m 0755 /dev/null wxyz ]
Executing command [ touch 0b00000001 ]
Executing command [ touch 0b00000010 ]
Executing command [ touch 0b00000011 ]
Executing command [ touch 0b00000100 ]
Executing command [ touch 0b00000101 ]
Executing command [ touch 0b00000110 ]
Executing command [ touch 0b00000111 ]
Executing command [ touch 0b00001000 ]
Executing command [ touch 0b00001001 ]
Executing command [ touch 0b00001010 ]
Executing command [ touch 0b00001011 ]
Executing command [ touch 0b00001100 ]
Executing command [ touch 0b00001101 ]
Executing command [ touch 0b00001110 ]
Executing command [ dd if=/dev/zero of=1.file bs=1 count=1 oseek=1 conv=sparse ]
Executing command [ dd if=/dev/zero of=512.file bs=1 count=1 oseek=512 conv=sparse ]
Executing command [ dd if=/dev/zero of=2048.file bs=1 count=1 oseek=2048 conv=sparse ]
Executing command [ dd if=/dev/zero of=10240.file bs=1 count=1 oseek=10240 conv=sparse ]
Executing command [ dd if=/dev/zero of=524288.file bs=1 count=1 oseek=524288 conv=sparse ]
Executing command [ makefs -t zfs -o nowarn=true -s 10g -o rootpath=/ -o poolname=makefstest30971 /tmp/kyua.78oQpz/3/work/test.img /tmp/kyua
.78oQpz/3/work/inputs ]
Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/3/work/test.img ]
Executing command [ zpool import -R /tmp/kyua.78oQpz/3/work/mnt makefstest30971 ]

Standard error:
Fail: incorrect exit status: 1, expected: 0
stdout:

stderr:
cannot import 'makefstest30971': no such pool available

Files left in work directory after failure: inputs, md.output, mnt, test.img
cat: /tmp/kyua.78oQpz/3/work/poolname: No such file or directory
cannot open '': name must begin with a letter
===> makefs_zfs_tests:empty_dir
Result:     failed: atf-check failed; see the output of the test for details
Start time: 2022-08-25T00:19:46.957465Z
End time:   2022-08-25T00:19:47.283094Z
Duration:   0.326s

Metadata:
    allowed_architectures is empty
    allowed_platforms is empty
    description is empty
    has_cleanup = true
    is_exclusive = false
    required_configs is empty
    required_disk_space = 0
    required_files is empty
    required_memory = 0
    required_programs is empty
    required_user is empty
    timeout = 300

Standard output:
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/5/work/mnt ]
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/5/work/inputs ]
Executing command [ makefs -t zfs -o nowarn=true -s 10g -o rootpath=/ -o poolname=makefstest30975 /tmp/kyua.78oQpz/5/work/test.img /tmp/kyua
.78oQpz/5/work/inputs ]
Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/5/work/test.img ]
Executing command [ zpool import -R /tmp/kyua.78oQpz/5/work/mnt makefstest30975 ]

Standard error:
Fail: incorrect exit status: 1, expected: 0
stdout:

stderr:
cannot import 'makefstest30975': pool already exists

Files left in work directory after failure: inputs, md.output, mnt, test.img
cat: /tmp/kyua.78oQpz/5/work/poolname: No such file or directory
cannot open '': name must begin with a letter
===> makefs_zfs_tests:empty_fs
Result:     failed: atf-check failed; see the output of the test for details
Start time: 2022-08-25T00:19:46.965942Z
End time:   2022-08-25T00:19:47.298847Z
Duration:   0.333s

Metadata:
    allowed_architectures is empty
    allowed_platforms is empty
    description is empty
    has_cleanup = true
    is_exclusive = false
    required_configs is empty
    required_disk_space = 0
    required_files is empty
    required_memory = 0
    required_programs is empty
    required_user is empty
    timeout = 300

Standard output:
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/6/work/mnt ]
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/6/work/inputs ]
Executing command [ makefs -t zfs -o nowarn=true -s 10g -o rootpath=/ -o poolname=makefstest30987 /tmp/kyua.78oQpz/6/work/test.img /tmp/kyua
.78oQpz/6/work/inputs ]
Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/6/work/test.img ]
Executing command [ zpool import -R /tmp/kyua.78oQpz/6/work/mnt makefstest30987 ]

Standard error:
Fail: incorrect exit status: 1, expected: 0
stdout:

stderr:
cannot import 'makefstest30987': no such pool available

Files left in work directory after failure: inputs, md.output, mnt, test.img
cat: /tmp/kyua.78oQpz/6/work/poolname: No such file or directory
cannot open '': name must begin with a letter
===> makefs_zfs_tests:file_sizes
Result:     failed: atf-check failed; see the output of the test for details
Start time: 2022-08-25T00:19:46.972400Z
End time:   2022-08-25T00:19:47.543610Z
Duration:   0.571s

Metadata:
    allowed_architectures is empty
    allowed_platforms is empty
    description is empty
    has_cleanup = true
    is_exclusive = false
    required_configs is empty
    required_disk_space = 0
    required_files is empty
    required_memory = 0
    required_programs is empty
    required_user is empty
    timeout = 300

Standard output:
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/7/work/mnt ]
Executing command [ mkdir -m 0777 -p /tmp/kyua.78oQpz/7/work/inputs ]
Executing command [ makefs -t zfs -o nowarn=true -s 10g -o rootpath=/ -o poolname=makefstest30998 /tmp/kyua.78oQpz/7/work/test.img /tmp/kyua
.78oQpz/7/work/inputs ]
Executing command [ mdconfig -a -f /tmp/kyua.78oQpz/7/work/test.img ]
Executing command [ zpool import -R /tmp/kyua.78oQpz/7/work/mnt makefstest30998 ]

Standard error:
Fail: incorrect exit status: 1, expected: 0
stdout:

stderr:
cannot import 'makefstest30998': no such pool available

Files left in work directory after failure: inputs, md.output, mnt, test.img
cat: /tmp/kyua.78oQpz/7/work/poolname: No such file or directory
cannot open '': name must begin with a letter

makefs_zfs regression tests are all using the same zpool name

The poolname contains the PID, so I guess kyua is executing multiple tests in the same process context? How can I see the full test stdout/stderr when running multiple tests in parallel?

Yes indeed, the PID are all diferrents so the poolname. The problem is elsewhere:

I see now. The problem is that pools created by makefs always have the same GUID, so they can't be imported simultaneously. I added a parameter to makefs to specify a PRNG seed, and if I modify the tests to use it, they can run in parallel without problems. So, I don't think it'll be necessary to serialize the makefs_zfs tests.

I see now. The problem is that pools created by makefs always have the same GUID, so they can't be imported simultaneously. I added a parameter to makefs to specify a PRNG seed, and if I modify the tests to use it, they can run in parallel without problems. So, I don't think it'll be necessary to serialize the makefs_zfs tests.

I tried several variants of this idea but was always able to trigger occasional collisions by running the ZFS tests in a loop overnight. So I ended up just committing a variant of this patch, together with some unrelated bug fixes.

I see now. The problem is that pools created by makefs always have the same GUID, so they can't be imported simultaneously. I added a parameter to makefs to specify a PRNG seed, and if I modify the tests to use it, they can run in parallel without problems. So, I don't think it'll be necessary to serialize the makefs_zfs tests.

I tried several variants of this idea but was always able to trigger occasional collisions by running the ZFS tests in a loop overnight. So I ended up just committing a variant of this patch, together with some unrelated bug fixes.

Thanks!