Many ports have bundled tests which are mostly not used by the framework. This is obviously disappointing as it takes a great chance to discover problems early from us. Tests could also make adoption of new architectures easier, as though arm and mips support is developing, not many people have actualy software and/or care to set up qemu-emulated jails to test their ports, while this could be done automatically.
Thus I propose to finally add full-fledged support for test target. Considerations:
- short test name; not sure where regression-test came from, but let's keep it short and simple
- complete target sequence similar to build/install: test-message, test-depends, pre-test, do-test, post-test. pre- do- post- targets overridable from individual makefiles as usual, default targets provided where possible
- perl framework already provides test target, I intend to be compatible with that; other frameworks/uses may do similar things
- there's TEST_TARGET knob for ports which use usual `make'; it behaves a bit differently than ALL_TARGET/INSTALL_TARGET: it's empty by default for which no do-test target is provided, but it can be set to a target name (upsteam tend to use different names like test or check), it which case ${MAKE_CMD} ${TEST_TARGET} is called with usual make env/args
- TEST_ENV, TEST_ARGS are provided defaulted to MAKE_ENV, MAKE_ARGS (compatible to what perl framework does); may be overridden by the port
- individial ports may always define their own do-test: target
- may add more standard tests later; for example, I can think of standard test for shared libs which tries to link with them. There are several known cases of broken shared libs in the tree, and stage-qa based check proved to give too many false positives
- test depends on install. It could just depend on build, but I want to make it more flexible - for example, test may expect files arranged in a target hierarchy, e.g. stagedir. Probably, it's also convenient because it brings in run-depends which may be needed for tests as well
However, note that this code is preliminary, draft and actually a huge copypasta. Careful review is needed. Unsolved issues:
- dependency handling; the Mk code for this is huge and I don't understand it completely
- what lists should TEST_DEPENDS be included into?
- what auxilitary targets are needed (pretty-print-test-depends? others)?
- should BUILD/RUN/LIB depends be added to TEST_DEPENDS at some point
- ...
- probably needs more documentation
- there are ports which need extra BUILD_DEPENDS for tests (see devel/pire). I'd like to handle these consistently, for which a dedicated OPTION should be defined (e.g. TESTS) and used in all ports with these conditions. This may be done separetely from this review though.
- poudriere interaction should be kept in mind. Obviously poudriere testport and poudriere bulk -t should run tests.