Most python modules are tested with pytest, so a lot of python ports set these same lines:
TEST_DEPENDS=${PYTHON_PKGNAMEPREFIX}pytest>=0:devel/py-pytest@${PY_FLAVOR}
do-test:
@cd ${TEST_WRKSRC} && ${SETENV} ${TEST_ENV} ${PYTHON_CMD} -m pytest -v -rsNot only this is a code duplication, but some ports also introduce unwanted inconsistencies, such as not setting TEST_ENV or missing arguments which produce verbose failure information.
So, introduce a Mk/Uses entry which allows enabling typical pytest testing with just USES+=pytest
Additionally, it currently supports:
- USES+=pytest:4 to use pytest 4 instead of the latest pytest (which is the process of update to 7.x, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256624)
- PYTEST_IGNORED_TESTS and PYTEST_BROKEN_TESTS variables used to mute failing tests (by automatically composing a -k expression to pytest). These work the same way, but are indented to convey slightly different meaning:
- PYTEST_IGNORED_TESTS is for tests which are not expected to pass (such as ones which require working database)
- PYTEST_BROKEN_TESTS is for tests which are broken and needs examination and fixing
Possible room for improvement:
- Plugins support (e.g. PYTEST_PLUGINS=asyncio cov) which automatically brings in depends, and disable unlisted plugins (which sometimes break testing) from pytest
- Better handling of PYTEST_IGNORED_TESTS/PYTEST_BROKEN_TESTS, which will probably require upstream changes. I'd expect the former to be treated by pytest as skipped, and the latter as xfail.
- Centralized handling of whether port must be installed or not before tests are run.