Change Details

Most python modules are tested with pytest, so a lot of python ports set these same lines: ``` TEST_DEPENDS=${PYTHON_PKGNAMEPREFIX}pytest>=0:devel/py-pytest@${PY_FLAVOR} do-test: @cd ${TEST_WRKSRC} && ${SETENV} ${TEST_ENV} ${PYTHON_CMD} -m pytest -v -rs ``` Not only this is a code duplication, but some ports also introduce unwanted inconsistencies, such as not setting TEST_ENV or missing arguments which produce verbose failure information. So, introduce a Mk/Uses entry which allows enabling typical pytest testing with just `USES+=pytest` Additionally, it currently supports: - `USES+=pytest:4` to use pytest 4 instead of the latest pytest (which is the process of update to 7.x, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256624) - `PYTEST_IGNORED_TESTS` and `PYTEST_BROKEN_TESTS` variables used to mute failing tests (by automatically composing a `-k` expression to pytest). These work the same way, but are indented to convey slightly different meaning: - `PYTEST_IGNORED_TESTS` is for tests which are not expected to pass (such as ones which require working database) - `PYTEST_BROKEN_TESTS` is for tests which are broken and needs examination and fixing Possible room for improvement: - Plugins support (e.g. `PYTEST_PLUGINS=asyncio cov`) which automatically brings in depends, and disable unlisted plugins (which sometimes break testing) from pytest - Better handling of `PYTEST_IGNORED_TESTS`/`PYTEST_BROKEN_TESTS`, which will probably require upstream changes. I'd expect the former to be treated by pytest as `skipped`, and the latter as `xfail`.

Most python modules are tested with pytest, so a lot of python ports set these same lines: ``` TEST_DEPENDS=${PYTHON_PKGNAMEPREFIX}pytest>=0:devel/py-pytest@${PY_FLAVOR} do-test: @cd ${TEST_WRKSRC} && ${SETENV} ${TEST_ENV} ${PYTHON_CMD} -m pytest -v -rs ``` Not only this is a code duplication, but some ports also introduce unwanted inconsistencies, such as not setting TEST_ENV or missing arguments which produce verbose failure information. So, introduce a Mk/Uses entry which allows enabling typical pytest testing with just `USES+=pytest` Additionally, it currently supports: - `USES+=pytest:4` to use pytest 4 instead of the latest pytest (which is the process of update to 7.x, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256624) - `PYTEST_IGNORED_TESTS` and `PYTEST_BROKEN_TESTS` variables used to mute failing tests (by automatically composing a `-k` expression to pytest). These work the same way, but are indented to convey slightly different meaning: - `PYTEST_IGNORED_TESTS` is for tests which are not expected to pass (such as ones which require working database) - `PYTEST_BROKEN_TESTS` is for tests which are broken and needs examination and fixing Possible room for improvement: - Plugins support (e.g. `PYTEST_PLUGINS=asyncio cov`) which automatically brings in depends, and disable unlisted plugins (which sometimes break testing) from pytest - Better handling of `PYTEST_IGNORED_TESTS`/`PYTEST_BROKEN_TESTS`, which will probably require upstream changes. I'd expect the former to be treated by pytest as `skipped`, and the latter as `xfail`. - Centralized handling of whether port must be installed or not before tests are run.

Most python modules are tested with pytest, so a lot of python ports set these same lines: ``` TEST_DEPENDS=${PYTHON_PKGNAMEPREFIX}pytest>=0:devel/py-pytest@${PY_FLAVOR} do-test: @cd ${TEST_WRKSRC} && ${SETENV} ${TEST_ENV} ${PYTHON_CMD} -m pytest -v -rs ``` Not only this is a code duplication, but some ports also introduce unwanted inconsistencies, such as not setting TEST_ENV or missing arguments which produce verbose failure information. So, introduce a Mk/Uses entry which allows enabling typical pytest testing with just `USES+=pytest` Additionally, it currently supports: - `USES+=pytest:4` to use pytest 4 instead of the latest pytest (which is the process of update to 7.x, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256624) - `PYTEST_IGNORED_TESTS` and `PYTEST_BROKEN_TESTS` variables used to mute failing tests (by automatically composing a `-k` expression to pytest). These work the same way, but are indented to convey slightly different meaning: - `PYTEST_IGNORED_TESTS` is for tests which are not expected to pass (such as ones which require working database) - `PYTEST_BROKEN_TESTS` is for tests which are broken and needs examination and fixing Possible room for improvement: - Plugins support (e.g. `PYTEST_PLUGINS=asyncio cov`) which automatically brings in depends, and disable unlisted plugins (which sometimes break testing) from pytest - Better handling of `PYTEST_IGNORED_TESTS`/`PYTEST_BROKEN_TESTS`, which will probably require upstream changes. I'd expect the former to be treated by pytest as `skipped`, and the latter as `xfail`. - Centralized handling of whether port must be installed or not before tests are run.