Page MenuHomeFreeBSD

Mk/Uses/pytest.mk: add Uses item for pytest
ClosedPublic

Authored by AMDmi3 on Mar 28 2022, 2:09 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Jun 16, 9:08 PM
Unknown Object (File)
Sun, Jun 16, 6:49 PM
Unknown Object (File)
Sun, Jun 16, 7:52 AM
Unknown Object (File)
Sun, Jun 16, 5:14 AM
Unknown Object (File)
Tue, Jun 11, 2:21 PM
Unknown Object (File)
May 25 2024, 11:32 AM
Unknown Object (File)
May 2 2024, 12:38 PM
Unknown Object (File)
Apr 28 2024, 1:46 PM

Details

Reviewers
None
Group Reviewers
portmgr
Commits
R11:b731fff69c0f: Mk/Uses: Add pytest.mk
Summary

Most python modules are tested with pytest, so a lot of python ports set these same lines:

TEST_DEPENDS=${PYTHON_PKGNAMEPREFIX}pytest>=0:devel/py-pytest@${PY_FLAVOR}

do-test:
   @cd ${TEST_WRKSRC} && ${SETENV} ${TEST_ENV} ${PYTHON_CMD} -m pytest -v -rs

Not only this is a code duplication, but some ports also introduce unwanted inconsistencies, such as not setting TEST_ENV or missing arguments which produce verbose failure information.

So, introduce a Mk/Uses entry which allows enabling typical pytest testing with just USES+=pytest

Additionally, it currently supports:

  • USES+=pytest:4 to use pytest 4 instead of the latest pytest (which is the process of update to 7.x, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256624)
  • PYTEST_IGNORED_TESTS and PYTEST_BROKEN_TESTS variables used to mute failing tests (by automatically composing a -k expression to pytest). These work the same way, but are indented to convey slightly different meaning:
    • PYTEST_IGNORED_TESTS is for tests which are not expected to pass (such as ones which require working database)
    • PYTEST_BROKEN_TESTS is for tests which are broken and needs examination and fixing

Possible room for improvement:

  • Plugins support (e.g. PYTEST_PLUGINS=asyncio cov) which automatically brings in depends, and disable unlisted plugins (which sometimes break testing) from pytest
  • Better handling of PYTEST_IGNORED_TESTS/PYTEST_BROKEN_TESTS, which will probably require upstream changes. I'd expect the former to be treated by pytest as skipped, and the latter as xfail.
  • Centralized handling of whether port must be installed or not before tests are run.
Test Plan

The patch includes switching 3 of my ports for this new framework to demonstrate the removal of duplicated code (saving 4 lines per typical pytest using port), and the use of new PYTEST_BROKEN_TESTS feature which makes the fact of failing tests visible, and allows remaining test to be used normally.

Diff Detail

Repository
R11 FreeBSD ports repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

AMDmi3 created this revision.
AMDmi3 edited the summary of this revision. (Show Details)
AMDmi3 edited the test plan for this revision. (Show Details)
tcberner added inline comments.
Uses/pytest.mk
2

Please indent the file using Tools/scripts/indent_make_if.pl.

8

^ would it not make more sense to error out in that case, as that is an error in the port?

Or are there non-python ports using pytest?

62

how about just adding a pytest-do-tes-target instead of using do-test, and add it via via _USES_test+= ?

py-asttokens/Makefile
20–24

maybe pytest.mk could automatically add the ^test_ to the names so that the list would be shorter ala

PYTEST_BROKEN_TESTS=	assignment_expressions async_def decorators [...]
AMDmi3 added inline comments.
Uses/pytest.mk
8

I forgot to change it, it should in fact be read python:test instead of python:build.

^ would it not make more sense to error out in that case, as that is an error in the port?

It would not. pytest is just an utility, so technically a port may use it without knowing anything about python. However, the ports framework does need to know about python because of multiple versions. So this should be totally transparent to ports.

Or are there non-python ports using pytest?

I haven't checked, but I don't see why there can't be.

62

I don't see what's it going to achieve. The indention is clearly to redefine a primary test target, as in we don't want to run pytest in addition to some default test target or any other predefined target. For instance, all the pytest users call it from do-test.

py-asttokens/Makefile
20–24

No, pytest.mk cannot do any assumptions about upstream test names, which may have no test_ prefix. Also the arguments are in fact patterns, which may start either before or after the test name. Because of that, we can just drop test_ prefix in the port and it will work, but I'd prefer not to and keep the full test names there.

AMDmi3 marked 2 inline comments as done.
  • Imply python:test instead of python:build when no USES=python is provided.
  • Fix formatting
AMDmi3 marked an inline comment as done.
  • Fix formatting (somehow missed from the previous revision)
  • Add -o addopts= which is also set by the most ports
This revision was not accepted when it landed; it landed in state Needs Review.Apr 18 2022, 10:42 AM
This revision was automatically updated to reflect the committed changes.