Page MenuHomeFreeBSD

Simplify the capsicum-test wrapper script
ClosedPublic

Authored by arichardson on Mar 2 2021, 5:27 PM.

Details

Summary

Instead of running tests one-by-one with the shell wrapper we now run
the full gtest testsuite twice (once as root, once as non root). This
significantly speeds up running tests despite running them twice.
This change also passes the missing -u flag to capsicum-test that caused
test failures (https://bugs.freebsd.org/250178)

Previously, running the testsuite with the wrapper script took ~3s per
test on aarch64 QEMU, i.e. a total of almost 5 minutes.
Now it takes 6 seconds to run all tests twice.

Before:
root@freebsd-aarch64:/usr/tests/sys/capsicum # /usr/bin/time kyua test functional
94/96 passed (2 failed)

309.97 real        58.46 user       244.31 sys

After:
root@freebsd-aarch64:/usr/tests/sys/capsicum # /usr/bin/time kyua test functional
functional:test_root -> passed [2.659s]
functional:test_unprivileged -> passed [2.391s]
2/2 passed (0 failed)

5.48 real         1.06 user         2.52 sys

This overhead is caused by kyua + atf-sh spawning lots of additional
processes and can be avoided by just running the googletest test binary.

truss -cf kyua test functional:Socket__UDP

syscall seconds calls errors
fork 39.810229456 1275 0
sigprocmask 13.546928736 572 0

i.e. 1275 processes spawned to run a single test.

PR: 250178

Test Plan

All tests pass after D28907.

Diff Detail

Repository
R10 FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

I still prefer to have test cases fine-grained as it is easier to track and debug, but since the performance difference is so large, it's not worthy to do this for now.

This revision is now accepted and ready to land.Mar 2 2021, 6:07 PM

I still prefer to have test cases fine-grained as it is easier to track and debug, but since the performance difference is so large, it's not worthy to do this for now.

I agree it would be nice, but that should probably be done in kyua and then we could run e.g. one suite at a time instead of each individual test.

This revision was landed with ongoing or failed builds.Mar 2 2021, 6:30 PM
This revision was automatically updated to reflect the committed changes.