Mk/Uses/pytest.mk: add Uses item for pytest

[AMDmi3]

Summary

Most python modules are tested with pytest, so a lot of python ports set these same lines:

TEST_DEPENDS=${PYTHON_PKGNAMEPREFIX}pytest>=0:devel/py-pytest@${PY_FLAVOR}

do-test:
   @cd ${TEST_WRKSRC} && ${SETENV} ${TEST_ENV} ${PYTHON_CMD} -m pytest -v -rs

Not only this is a code duplication, but some ports also introduce unwanted inconsistencies, such as not setting TEST_ENV or missing arguments which produce verbose failure information.

So, introduce a Mk/Uses entry which allows enabling typical pytest testing with just USES+=pytest

Additionally, it currently supports:

• USES+=pytest:4 to use pytest 4 instead of the latest pytest (which is the process of update to 7.x, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256624)
• PYTEST_IGNORED_TESTS and PYTEST_BROKEN_TESTS variables used to mute failing tests (by automatically composing a -k expression to pytest). These work the same way, but are indented to convey slightly different meaning:
  • PYTEST_IGNORED_TESTS is for tests which are not expected to pass (such as ones which require working database)
  • PYTEST_BROKEN_TESTS is for tests which are broken and needs examination and fixing

Possible room for improvement:

• Plugins support (e.g. PYTEST_PLUGINS=asyncio cov) which automatically brings in depends, and disable unlisted plugins (which sometimes break testing) from pytest
• Better handling of PYTEST_IGNORED_TESTS/PYTEST_BROKEN_TESTS, which will probably require upstream changes. I'd expect the former to be treated by pytest as skipped, and the latter as xfail.
• Centralized handling of whether port must be installed or not before tests are run.

Test Plan

The patch includes switching 3 of my ports for this new framework to demonstrate the removal of duplicated code (saving 4 lines per typical pytest using port), and the use of new PYTEST_BROKEN_TESTS feature which makes the fact of failing tests visible, and allows remaining test to be used normally.

[tcberner]

Uses/pytest.mk

2

Please indent the file using Tools/scripts/indent_make_if.pl.

8

^ would it not make more sense to error out in that case, as that is an error in the port?

Or are there non-python ports using pytest?

62

how about just adding a pytest-do-tes-target instead of using do-test, and add it via via _USES_test+= ?

py-asttokens/Makefile

20–24

maybe pytest.mk could automatically add the ^test_ to the names so that the list would be shorter ala

PYTEST_BROKEN_TESTS=	assignment_expressions async_def decorators [...]

[AMDmi3]

Uses/pytest.mk

8

I forgot to change it, it should in fact be read python:test instead of python:build.

^ would it not make more sense to error out in that case, as that is an error in the port?

It would not. pytest is just an utility, so technically a port may use it without knowing anything about python. However, the ports framework does need to know about python because of multiple versions. So this should be totally transparent to ports.

Or are there non-python ports using pytest?

I haven't checked, but I don't see why there can't be.

62

I don't see what's it going to achieve. The indention is clearly to redefine a primary test target, as in we don't want to run pytest in addition to some default test target or any other predefined target. For instance, all the pytest users call it from do-test.

py-asttokens/Makefile

20–24

No, pytest.mk cannot do any assumptions about upstream test names, which may have no test_ prefix. Also the arguments are in fact patterns, which may start either before or after the test name. Because of that, we can just drop test_ prefix in the port and it will work, but I'd prefer not to and keep the full test names there.

[AMDmi3]

• Imply python:test instead of python:build when no USES=python is provided.
• Fix formatting

[AMDmi3]

• Fix formatting (somehow missed from the previous revision)
• Add -o addopts= which is also set by the most ports

[AMDmi3]

@tcberner ping? does it look good now?