Update devel/kyua to 0.12.
rP402256
Actions

Description

Update devel/kyua to 0.12.

Released on November 22nd, 2015.

This is a huge release and marks a major milestone for Kyua as it finally
implements a long-standing feature request: the ability to execute test
cases in parallel. This is a big deal because test cases are rarely
CPU-bound: running them in parallel yields much faster execution times for
large test suites, allowing faster iteration of changes during development.

As an example: the FreeBSD test suite as of this date contains 3285 test
cases. With sequential execution, a full test suite run takes around 12
minutes to complete, whereas on a 4-core machine with a high level of
parallelism it takes a little over 1 minute.

Implementing parallel execution required rewriting most of Kyua's core and
partly explains explains why there has not been a new release for over a
year. The current implementation is purely subprocess-based, which works
but has some limitations and has resulted in a core that is really complex
and difficult to understand. Future versions will investigate the use of
threads instead for a simplified programming model and additional
parallelization possibilities.

Issue #2: Implemented support to execute test cases in parallel when invoking kyua test. Parallel execution is *only* enabled when the new parallelism configuration variable is set to a value greater than 1. The default behavior is still to run tests sequentially because some test suites contain test cases with side-effects that might fail when run in parallel. To resolve this, the new metadata property is_exclusive can be set to true on a test basis to indicate that the test must be run on its own.

Known regression: Running kyua debug on a TAP-based test program does not currently report the output in real time. The output will only be displayed once the test program completes. This is a shortcoming of the new parallel execution engine and will be resolved.

Removed the external C-based testers code in favor of the new built-in implementations. The new approach feels significantly faster than the previous one.

Fixed the handling of relative paths in the fs.* functions available in Kyuafiles. All paths are now resolved relative to the location of the caller Kyuafile. Kyuafile.top has been updated with these changes and you should update custom copies of this file with the new version.

Changed temporary directory creation to always grant search permissions on temporary directories. This is to prevent potential problems when running Kyua as root and executing test cases that require dropping privileges (as they may later be unable to use absolute paths that point inside their work directory).

The cleanup of work directories does not longer attempt to deal with mount points. If a test case mounts a file system and forgets to unmount it, the mount point will be left behind. It is now the responsibility of the test case to clean after itself. The reasons for this change are simplicity and clarity: there are many more things that a test case can do that have side-effects on the system and Kyua cannot protect against them all, so it is better to just have the test undo anything it might have done.

Improved kyua report --verbose to properly handle environment variables with continuation lines in them, and fixed the integration tests for this command to avoid false negatives.

Changed the configuration file format to accept the definition of unknown variables without declaring them local. The syntax version number remains at 2. This is to allow configuration files for newer Kyua versions to work on older Kyua versions, as there is no reason to forbid this.

Fixed stacktrace gathering with FreeBSD's ancient version of GDB. GDB 6.1.1 (circa 2004) does not have the -ex flag so we need to generate a temporary GDB script and feed it to GDB with -x instead.

Issue #136: Fixed the XML escaping in the JUnit output so that non-printable characters are properly handled when they appear in the process's stdout or stderr.

Issue #141: Improved reporting of errors triggered by sqlite3. In particular, all error messages are now tagged with their corresponding database filename and, if they are API-level errors, the name of the sqlite3 function that caused them.