HomeFreeBSD

Mk/Uses: Add pytest.mk

Description

Mk/Uses: Add pytest.mk

Add USES=pytest, a standardized pytest entry point.

This knob adds dependency on pytest and defines do-test target which
calls it "the right way" (respecting TEST_ENV, with proper verbosity,
additional info on skipped tests, and resetting addopts which may
be set by upstream and introduce unwanted options). This allows to
remove duplicate (and in many cases not completely correct) do-test
targets from a lot of pytest consumer ports.

Additionally, it adds PYTEST_IGNORED_TESTS (to list tests which
fail and are expected to fail) and PYTEST_BROKEN_TESTS (to list
tests which fail and needs to be fixed). Both knobs lead to
listed tests being skipped, which is believed to be better than
always failing test target, allowing to monitor status of other
tests and better document known failures.

Possible room for improvement:

  • Support for pytest plugins (add dependencies and disallow pytest from using unlisted plugins which often cause breakages)
  • Treat PYTEST_BROKEN_TESTS as xfail to catch the cases when tests are fixed.

    Both items seem to require support on pytest side though.

Reviewed by: tcberner
Differential Revision: https://reviews.freebsd.org/D34693

Details

Provenance
AMDmi3Authored on Apr 15 2022, 2:17 PM
Reviewer
tcberner
Differential Revision
D34693: Mk/Uses/pytest.mk: add Uses item for pytest
Parents
R11:1cfe12ed25e5: devel/ocaml-ocamlbuild: update the port to version 1.14.1
Branches
Unknown
Tags
Unknown