Page MenuHomeFreeBSD

[NEW PORT] devel/py-flaky: Plugin for nose or pytest that automatically reruns flaky tests
ClosedPublic

Authored by kai on Aug 2 2019, 9:12 AM.

Details

Reviewers
koobs
Group Reviewers
Python
Commits
rP508128: [NEW PORT] devel/py-flaky
Summary
[NEW PORT] devel/py-flaky: Plugin for nose or pytest that automatically reruns flaky tests

Ideally, tests reliably pass or fail, but sometimes test fixtures must rely on
components that aren't 100% reliable.

With flaky, instead of removing those tests or marking them to skip, they can
be automatically retried.

WWW: https://github.com/box/flaky

Reviewed_by: koobs
Approved by: koobs
Differential_Revision: D21132
Test Plan
  • testport: OK (poudriere: 11.2-, 11.3-, 12.0-RELEASE, 13.0-CURRENT@r349352 amd64 i386) for each py27 + py36 flavor -> OK
  • portlint OK
  • maketest: OK

Questions/notes from my side:

  • Do we need that many tests (all taken from upstream's tox.ini except the one with "--doctest-modules" in it)?
  • Is there a better way to compact/handle all those tests to fewer lines?

The commands in the "do-test" target are formatted by the following convention:
'''
${PYTHON_CMD} -m ${OWN_ARGS} && \
${UPSTREAM_ARGS}
'''

The argument "--no-flaky-report" was added to avoid cluttering of the test results.

Diff Detail

Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 25637
Build 24230: arc lint + arc unit

Event Timeline

kai created this revision.Aug 2 2019, 9:12 AM
koobs retitled this revision from [New port]: devel/py-flaky: Plugin for nose or pytest that automatically reruns flaky tests to [NEW PORT] devel/py-flaky: Plugin for nose or pytest that automatically reruns flaky tests.Aug 3 2019, 2:32 AM
koobs edited the test plan for this revision. (Show Details)
koobs accepted this revision.Aug 3 2019, 2:54 AM
koobs edited the summary of this revision. (Show Details)
koobs added a subscriber: koobs.

If the port installs files in shared (for multiple Python verion) locations, port must be concurrent safe

devel/py-flaky/Makefile
26

setup.py defines test_suite='test' which sets up/enables the setup.py test command. What happens when that's run Does it run all tests?

28–37

Yeh that's annoying, but its what tox.ini runs.

In the absence of a single command to run all tests, then we can only execute what upstream defines

Ultimately, a single pytest / nosetest / setup.py test should just run all the tests, using the build in test discovery mechanisms to find all tests recursively in test/*.

Tool specific options arguments should go in setup.cfg

This revision is now accepted and ready to land.Aug 3 2019, 2:55 AM
kai added a comment.Aug 3 2019, 9:01 AM

If the port installs files in shared (for multiple Python verion) locations, port must be concurrent safe

Thanks for the heads-up, I double checked it and can say that the port only installs files into ${PYTHON_SITELIBDIR}.

devel/py-flaky/Makefile
26

No, it just throws an exception:

Traceback (most recent call last):
  File "setup.py", line 85, in <module>
    main()
  File "setup.py", line 80, in main
    classifiers=CLASSIFIERS,
  File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup                                                                                  
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.6/distutils/core.py", line 134, in setup
    ok = dist.parse_command_line()
  File "/usr/local/lib/python3.6/site-packages/setuptools/dist.py", line 703, in parse_command_line                                                                         
    result = _Distribution.parse_command_line(self)
  File "/usr/local/lib/python3.6/distutils/dist.py", line 472, in parse_command_line
    args = self._parse_command_opts(parser, args)
  File "/usr/local/lib/python3.6/site-packages/setuptools/dist.py", line 1018, in _parse_command_opts                                                                       
    nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/usr/local/lib/python3.6/distutils/dist.py", line 567, in _parse_command_opts
    (args, opts) = parser.getopt(args[1:])
  File "/usr/local/lib/python3.6/distutils/fancy_getopt.py", line 229, in getopt
    self._grok_option_table()
  File "/usr/local/lib/python3.6/distutils/fancy_getopt.py", line 157, in _grok_option_table                                                                                
    "must be a string of length >= 2") % long)
distutils.errors.DistutilsGetoptError: invalid long option 'b'tox-args='': must be a string of length >= 2
This revision was automatically updated to reflect the committed changes.