[NEW PORT] devel/py-flaky: Plugin for nose or pytest that automatically reruns flaky tests

[kai]

Summary

[NEW PORT] devel/py-flaky: Plugin for nose or pytest that automatically reruns flaky tests

Ideally, tests reliably pass or fail, but sometimes test fixtures must rely on
components that aren't 100% reliable.

With flaky, instead of removing those tests or marking them to skip, they can
be automatically retried.

WWW: https://github.com/box/flaky

Reviewed_by: koobs
Approved by: koobs
Differential_Revision: D21132

Test Plan

• testport: OK (poudriere: 11.2-, 11.3-, 12.0-RELEASE, 13.0-CURRENT@r349352 amd64 i386) for each py27 + py36 flavor -> OK
• portlint OK
• maketest: OK

Questions/notes from my side:

• Do we need that many tests (all taken from upstream's tox.ini except the one with "--doctest-modules" in it)?
• Is there a better way to compact/handle all those tests to fewer lines?

The commands in the "do-test" target are formatted by the following convention:
'''
${PYTHON_CMD} -m ${OWN_ARGS} && \
${UPSTREAM_ARGS}
'''

The argument "--no-flaky-report" was added to avoid cluttering of the test results.

[koobs]

If the port installs files in shared (for multiple Python verion) locations, port must be concurrent safe

devel/py-flaky/Makefile

26 ↗

(On Diff #60382)

setup.py defines test_suite='test' which sets up/enables the setup.py test command. What happens when that's run Does it run all tests?

28–37 ↗

(On Diff #60382)

Yeh that's annoying, but its what tox.ini runs.

In the absence of a single command to run all tests, then we can only execute what upstream defines

Ultimately, a single pytest / nosetest / setup.py test should just run all the tests, using the build in test discovery mechanisms to find all tests recursively in test/*.

Tool specific options arguments should go in setup.cfg

[kai]

In D21132#459422, @koobs wrote:

If the port installs files in shared (for multiple Python verion) locations, port must be concurrent safe

Thanks for the heads-up, I double checked it and can say that the port only installs files into ${PYTHON_SITELIBDIR}.

devel/py-flaky/Makefile

26 ↗

(On Diff #60382)

No, it just throws an exception:

Traceback (most recent call last):
  File "setup.py", line 85, in <module>
    main()
  File "setup.py", line 80, in main
    classifiers=CLASSIFIERS,
  File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup                                                                                  
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.6/distutils/core.py", line 134, in setup
    ok = dist.parse_command_line()
  File "/usr/local/lib/python3.6/site-packages/setuptools/dist.py", line 703, in parse_command_line                                                                         
    result = _Distribution.parse_command_line(self)
  File "/usr/local/lib/python3.6/distutils/dist.py", line 472, in parse_command_line
    args = self._parse_command_opts(parser, args)
  File "/usr/local/lib/python3.6/site-packages/setuptools/dist.py", line 1018, in _parse_command_opts                                                                       
    nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/usr/local/lib/python3.6/distutils/dist.py", line 567, in _parse_command_opts
    (args, opts) = parser.getopt(args[1:])
  File "/usr/local/lib/python3.6/distutils/fancy_getopt.py", line 229, in getopt
    self._grok_option_table()
  File "/usr/local/lib/python3.6/distutils/fancy_getopt.py", line 157, in _grok_option_table                                                                                
    "must be a string of length >= 2") % long)
distutils.errors.DistutilsGetoptError: invalid long option 'b'tox-args='': must be a string of length >= 2