A lot of ports use setuptools, some have test command defined. In order to avoid redundancy and minor differences let's streamline testing. For details see individual commit messages.
Details
Details
Only very few ports tested manually on a single architecture/release combination e.g.,
# pkg delete -ayq pkg: Cannot delete pkg itself without force flag # make test -C devel/py-apscheduler ===> License MIT accepted by the user ===> py27-apscheduler-3.0.3 depends on file: /usr/local/sbin/pkg - found ===> Fetching all distfiles required by py27-apscheduler-3.0.3 for building ===> Extracting for py27-apscheduler-3.0.3 => SHA256 Checksum OK for APScheduler-3.0.3.tar.gz. ===> Patching for py27-apscheduler-3.0.3 ===> py27-apscheduler-3.0.3 depends on package: py27-setuptools27>0 - not found ===> Installing existing package /packages/All/py27-setuptools27-17.0.txz [93i386-default] Installing py27-setuptools27-17.0... [93i386-default] `-- Installing python27-2.7.10... [93i386-default] | `-- Installing libffi-3.2.1... [93i386-default] | | `-- Installing indexinfo-0.2.3... [93i386-default] | | `-- Extracting indexinfo-0.2.3: 100% [93i386-default] | `-- Extracting libffi-3.2.1: 100% [93i386-default] | `-- Installing gettext-runtime-0.19.5.1... [93i386-default] | | `-- Installing libiconv-1.14_8... [93i386-default] | | `-- Extracting libiconv-1.14_8: 100% [93i386-default] | `-- Extracting gettext-runtime-0.19.5.1: 100% [93i386-default] `-- Extracting python27-2.7.10: 100% [93i386-default] Extracting py27-setuptools27-17.0: 100% Message from python27-2.7.10: ===================================================================== Note that some standard Python modules are provided as separate ports as they require additional dependencies. They are available as: bsddb databases/py-bsddb gdbm databases/py-gdbm sqlite3 databases/py-sqlite3 tkinter x11-toolkits/py-tkinter ===================================================================== ===> py27-apscheduler-3.0.3 depends on package: py27-setuptools27>0 - found ===> Returning to build of py27-apscheduler-3.0.3 ===> py27-apscheduler-3.0.3 depends on file: /usr/local/bin/python2.7 - found ===> Configuring for py27-apscheduler-3.0.3 running config ===> Building for py27-apscheduler-3.0.3 running build running build_py creating build creating build/lib creating build/lib/apscheduler copying apscheduler/util.py -> build/lib/apscheduler copying apscheduler/__init__.py -> build/lib/apscheduler copying apscheduler/events.py -> build/lib/apscheduler copying apscheduler/job.py -> build/lib/apscheduler creating build/lib/apscheduler/schedulers copying apscheduler/schedulers/tornado.py -> build/lib/apscheduler/schedulers copying apscheduler/schedulers/__init__.py -> build/lib/apscheduler/schedulers copying apscheduler/schedulers/background.py -> build/lib/apscheduler/schedulers copying apscheduler/schedulers/twisted.py -> build/lib/apscheduler/schedulers copying apscheduler/schedulers/blocking.py -> build/lib/apscheduler/schedulers copying apscheduler/schedulers/base.py -> build/lib/apscheduler/schedulers copying apscheduler/schedulers/qt.py -> build/lib/apscheduler/schedulers copying apscheduler/schedulers/gevent.py -> build/lib/apscheduler/schedulers copying apscheduler/schedulers/asyncio.py -> build/lib/apscheduler/schedulers creating build/lib/apscheduler/jobstores copying apscheduler/jobstores/memory.py -> build/lib/apscheduler/jobstores copying apscheduler/jobstores/base.py -> build/lib/apscheduler/jobstores copying apscheduler/jobstores/__init__.py -> build/lib/apscheduler/jobstores copying apscheduler/jobstores/redis.py -> build/lib/apscheduler/jobstores copying apscheduler/jobstores/mongodb.py -> build/lib/apscheduler/jobstores copying apscheduler/jobstores/sqlalchemy.py -> build/lib/apscheduler/jobstores creating build/lib/apscheduler/triggers copying apscheduler/triggers/__init__.py -> build/lib/apscheduler/triggers copying apscheduler/triggers/base.py -> build/lib/apscheduler/triggers copying apscheduler/triggers/date.py -> build/lib/apscheduler/triggers copying apscheduler/triggers/interval.py -> build/lib/apscheduler/triggers creating build/lib/apscheduler/executors copying apscheduler/executors/__init__.py -> build/lib/apscheduler/executors copying apscheduler/executors/pool.py -> build/lib/apscheduler/executors copying apscheduler/executors/twisted.py -> build/lib/apscheduler/executors copying apscheduler/executors/asyncio.py -> build/lib/apscheduler/executors copying apscheduler/executors/debug.py -> build/lib/apscheduler/executors copying apscheduler/executors/gevent.py -> build/lib/apscheduler/executors copying apscheduler/executors/base.py -> build/lib/apscheduler/executors creating build/lib/apscheduler/triggers/cron copying apscheduler/triggers/cron/fields.py -> build/lib/apscheduler/triggers/cron copying apscheduler/triggers/cron/expressions.py -> build/lib/apscheduler/triggers/cron copying apscheduler/triggers/cron/__init__.py -> build/lib/apscheduler/triggers/cron ===> Staging for py27-apscheduler-3.0.3 ===> py27-apscheduler-3.0.3 depends on package: py27-pytz>0 - not found ===> Installing existing package /packages/All/py27-pytz-2015.4,1.txz [93i386-default] Installing py27-pytz-2015.4,1... [93i386-default] Extracting py27-pytz-2015.4,1: 100% ===> py27-apscheduler-3.0.3 depends on package: py27-pytz>0 - found ===> Returning to build of py27-apscheduler-3.0.3 ===> py27-apscheduler-3.0.3 depends on package: py27-six>=1.4.0 - not found ===> Installing existing package /packages/All/py27-six-1.9.0.txz [93i386-default] Installing py27-six-1.9.0... [93i386-default] Extracting py27-six-1.9.0: 100% ===> py27-apscheduler-3.0.3 depends on package: py27-six>=1.4.0 - found ===> Returning to build of py27-apscheduler-3.0.3 ===> py27-apscheduler-3.0.3 depends on package: py27-tzlocal>0 - not found ===> Installing existing package /packages/All/py27-tzlocal-1.1.1.txz [93i386-default] Installing py27-tzlocal-1.1.1... [93i386-default] Extracting py27-tzlocal-1.1.1: 100% ===> py27-apscheduler-3.0.3 depends on package: py27-tzlocal>0 - found ===> Returning to build of py27-apscheduler-3.0.3 ===> py27-apscheduler-3.0.3 depends on package: py27-setuptools27>0 - found ===> py27-apscheduler-3.0.3 depends on file: /usr/local/bin/python2.7 - found ===> py27-apscheduler-3.0.3 depends on package: py27-futures>0 - not found ===> Installing existing package /packages/All/py27-futures-3.0.3.txz [93i386-default] Installing py27-futures-3.0.3... [93i386-default] `-- Installing python2-2_3... [93i386-default] `-- Extracting python2-2_3: 100% [93i386-default] Extracting py27-futures-3.0.3: 100% ===> py27-apscheduler-3.0.3 depends on package: py27-futures>0 - found ===> Returning to build of py27-apscheduler-3.0.3 ===> Generating temporary packing list running install running build running build_py running install_lib creating /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7 creating /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages creating /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler copying build/lib/apscheduler/__init__.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler creating /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores copying build/lib/apscheduler/jobstores/sqlalchemy.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores copying build/lib/apscheduler/jobstores/redis.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores copying build/lib/apscheduler/jobstores/__init__.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores copying build/lib/apscheduler/jobstores/base.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores copying build/lib/apscheduler/jobstores/memory.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores copying build/lib/apscheduler/jobstores/mongodb.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores copying build/lib/apscheduler/util.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler copying build/lib/apscheduler/events.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler copying build/lib/apscheduler/job.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler creating /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers copying build/lib/apscheduler/triggers/base.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers copying build/lib/apscheduler/triggers/interval.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers copying build/lib/apscheduler/triggers/date.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers creating /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/cron copying build/lib/apscheduler/triggers/cron/fields.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/cron copying build/lib/apscheduler/triggers/cron/__init__.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/cron copying build/lib/apscheduler/triggers/cron/expressions.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/cron copying build/lib/apscheduler/triggers/__init__.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers creating /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers copying build/lib/apscheduler/schedulers/tornado.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers copying build/lib/apscheduler/schedulers/blocking.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers copying build/lib/apscheduler/schedulers/qt.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers copying build/lib/apscheduler/schedulers/gevent.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers copying build/lib/apscheduler/schedulers/base.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers copying build/lib/apscheduler/schedulers/asyncio.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers copying build/lib/apscheduler/schedulers/background.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers copying build/lib/apscheduler/schedulers/twisted.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers copying build/lib/apscheduler/schedulers/__init__.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers creating /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors copying build/lib/apscheduler/executors/base.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors copying build/lib/apscheduler/executors/twisted.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors copying build/lib/apscheduler/executors/asyncio.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors copying build/lib/apscheduler/executors/debug.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors copying build/lib/apscheduler/executors/gevent.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors copying build/lib/apscheduler/executors/pool.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors copying build/lib/apscheduler/executors/__init__.py -> /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/__init__.py to __init__.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy.py to sqlalchemy.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores/redis.py to redis.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores/__init__.py to __init__.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores/base.py to base.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores/memory.py to memory.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/jobstores/mongodb.py to mongodb.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/util.py to util.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/events.py to events.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/job.py to job.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/base.py to base.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/interval.py to interval.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/date.py to date.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/cron/fields.py to fields.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/cron/__init__.py to __init__.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/cron/expressions.py to expressions.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/triggers/__init__.py to __init__.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers/tornado.py to tornado.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers/blocking.py to blocking.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers/qt.py to qt.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers/gevent.py to gevent.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers/base.py to base.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers/asyncio.py to asyncio.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers/background.py to background.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers/twisted.py to twisted.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/schedulers/__init__.py to __init__.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors/base.py to base.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors/twisted.py to twisted.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors/asyncio.py to asyncio.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors/debug.py to debug.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors/gevent.py to gevent.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors/pool.py to pool.pyc byte-compiling /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/apscheduler/executors/__init__.py to __init__.pyc writing byte-compilation script '/tmp/tmp07JgRR.py' /usr/local/bin/python2.7 -O /tmp/tmp07JgRR.py removing /tmp/tmp07JgRR.py running install_egg_info running egg_info writing requirements to APScheduler.egg-info/requires.txt writing APScheduler.egg-info/PKG-INFO writing top-level names to APScheduler.egg-info/top_level.txt writing dependency_links to APScheduler.egg-info/dependency_links.txt writing entry points to APScheduler.egg-info/entry_points.txt reading manifest file 'APScheduler.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'APScheduler.egg-info/SOURCES.txt' Copying APScheduler.egg-info to /wrkdirs/usr/ports/devel/py-apscheduler/work/stage/usr/local/lib/python2.7/site-packages/APScheduler-3.0.3-py2.7.egg-info running install_scripts writing list of installed files to '/wrkdirs/usr/ports/devel/py-apscheduler/work/.PLIST.pymodtmp' ====> Compressing man pages (compress-man) ===> Testing for py27-apscheduler-3.0.3 ===> py27-apscheduler-3.0.3 depends on package: py27-pytest>=2.5.1 - not found ===> Installing existing package /packages/All/py27-pytest-2.7.1.txz [93i386-default] Installing py27-pytest-2.7.1... [93i386-default] `-- Installing py27-py-1.4.26... [93i386-default] `-- Extracting py27-py-1.4.26: 100% [93i386-default] Extracting py27-pytest-2.7.1: 100% ===> py27-apscheduler-3.0.3 depends on package: py27-pytest>=2.5.1 - found ===> Returning to build of py27-apscheduler-3.0.3 ===> py27-apscheduler-3.0.3 depends on package: py27-mock>0 - not found ===> Installing existing package /packages/All/py27-mock-1.0.1_1.txz [93i386-default] Installing py27-mock-1.0.1_1... [93i386-default] Extracting py27-mock-1.0.1_1: 100% ===> py27-apscheduler-3.0.3 depends on package: py27-mock>0 - found ===> Returning to build of py27-apscheduler-3.0.3 running test running egg_info writing requirements to APScheduler.egg-info/requires.txt writing APScheduler.egg-info/PKG-INFO writing top-level names to APScheduler.egg-info/top_level.txt writing dependency_links to APScheduler.egg-info/dependency_links.txt writing entry points to APScheduler.egg-info/entry_points.txt reading manifest file 'APScheduler.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'APScheduler.egg-info/SOURCES.txt' running build_ext =============================== test session starts ================================ platform freebsd9 -- Python 2.7.10 -- py-1.4.26 -- pytest-2.7.1 rootdir: /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3, inifile: setup.cfg collected 401 items tests/test_executors.py .......... tests/test_expressions.py .......................... tests/test_job.py ......................................... tests/test_jobstores.py .sss.sss.sss.sss.sss.sss.sss.sss..ssssss......ssssssssssssssssss.ssssss.sss.sss.sss.sss.sssss tests/test_schedulers.py .....................................................................................................sssssssssssssss tests/test_triggers.py ........................................ tests/test_util.py ...................................................ss....................ss ============================= short test summary info ============================== SKIP [1] /usr/local/lib/python2.7/site-packages/_pytest/skipping.py:140: Requires Python >= 3 SKIP [1] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_jobstores.py:286: could not import 'apscheduler.jobstores.mongodb' SKIP [3] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_schedulers.py:990: could not import 'tornado.ioloop' SKIP [1] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_jobstores.py:269: could not import 'apscheduler.jobstores.sqlalchemy' SKIP [3] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_schedulers.py:1015: could not import 'twisted.internet.selectreactor' SKIP [1] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_jobstores.py:296: could not import 'apscheduler.jobstores.mongodb' SKIP [3] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_schedulers.py:971: could not import 'apscheduler.schedulers.gevent' SKIP [3] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_schedulers.py:1041: could not import 'PySide.QtCore' SKIP [22] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_jobstores.py:58: could not import 'apscheduler.jobstores.redis' SKIP [3] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_schedulers.py:946: could not import 'apscheduler.schedulers.asyncio' SKIP [1] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_jobstores.py:279: could not import 'apscheduler.jobstores.sqlalchemy' SKIP [22] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_jobstores.py:46: could not import 'apscheduler.jobstores.mongodb' SKIP [2] /usr/local/lib/python2.7/site-packages/_pytest/skipping.py:140: Requires __qualname__ (Python 3.3+) SKIP [22] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_jobstores.py:34: could not import 'apscheduler.jobstores.sqlalchemy' SKIP [1] /wrkdirs/usr/ports/devel/py-apscheduler/work/APScheduler-3.0.3/tests/test_jobstores.py:302: could not import 'apscheduler.jobstores.mongodb' SKIP [1] /usr/local/lib/python2.7/site-packages/_pytest/skipping.py:140: Requires Python >= 3.4 ====================== 311 passed, 90 skipped in 3.71 seconds ======================
Diff Detail
Diff Detail
- Repository
- rP FreeBSD ports repository
- Lint
No Lint Coverage - Unit
No Test Coverage - Build Status
Buildable 671 Build 671: arc lint + arc unit
Event Timeline
Comment Actions
(only reviewed a few bits, will have a time to look through remaining part later)
| Mk/bsd.port.mk | ||
|---|---|---|
| 1510 ↗ | (On Diff #9099) | Please don't. This way you have less control and it's also inconsistent with _ARGS. For instance, if for some reason you want TEST_ENV empty, with your change you can't make it so unless you also make MAKE_ENV empty, and you may not want that as you need MAKE_ENV for build phase. With ?= you can do whatever you need - you can set TEST_ENV it to empty, to ${MAKE_ENV}, to foo ${MAKE_ENV} or to ${MAKE_ENV} foo for any kind of override logic. |
| databases/py-Pyrseas/Makefile | ||
| 25–28 | I think you need NO_TEST here, else the test will always fail. Not yet sure, but either
| |
Comment Actions
- bsd.port.mk patch dropped as it needs more work but ENOTIME. TEST_ENV= ${MAKE_ENV} ... should suffice here to run setup.py commands with consistent environment. Still a shot in the dark without automation help.
- Adapted databases/py-Pyrseas to not run for PACKAGE_BUILDING. However, more ports here have broken tests. Identifying them is outside of scope until poudriere catches up. Their tests can be disabled individually with errors documented in commit messages. regression-test implies it worked fine in the past as tinderbox runs the target by default.
Comment Actions
I'm -1 on this so soon after the test framework landed. I think a review of the current test framework for applicability to Python ports is warranted before we go changing a large number of Python ports.
See:
| Mk/Uses/python.mk | ||
|---|---|---|
| 118 | I would prefer to see this as a generic variable, overriding the default TEST_TARGET from bsd.port.mk. The canonical 'test' target is a setuptools target, so naming it 'distutils' is technically incorrect. Also, my gut feeling tells me we want to explicitly opt-in/enable tests, indepently to whether other TEST_* variables are set. Many upstream test suites break/fail from one version to the next. | |
Comment Actions
Let's catch up on #freebsd-python (freenode IRC) so we can discuss and elicit feedback from others
Comment Actions
Sorry, that was an accident. I've intentionally moved portmgr from reviewers to subscribers as the patch against bsd.port.mk is gone in new version.
| Mk/Uses/python.mk | ||
|---|---|---|
| 118 |
That will be inconsistent with other PYDISTUTILS_*_TARGETs. TEST_TARGET is intended primarily for make, so there's nothing wrong if different kind of target variables are used for pydistutils. Otoh, we may consider switching these to normal *_TARGETs.
I'd say opt-out. We want tests to run by default, and disable them only if they are useless and cannot be fixed. Every failure should be fixed this way or another, as it's either problem of software or test itself. | |
| Mk/Uses/python.mk | ||
|---|---|---|
| 118 |
How existing regression-test in the py-* ports survived tinderbox run then? In pointyhat.freebsd.org days such ports would be quickly flagged as BROKEN or the target would be disabled somehow. Since poudriere became the tool of choice testing automation gradually regressed. Until it catches up tests are unreliable. They can fail on some cpython versions, different freebsd releases or architectures. Only maintainer knows where or when tests are supposed to work. Otherwise, NO_TEST works fine as a tool to disable broken tests without also disabling package generation. | |
| Mk/Uses/python.mk | ||
|---|---|---|
| 610 | How about ".if !target(do-test) && !defined(NO_PYTHON_TEST)"? I would suggest to add check for NO_PYTHON_TEST. It allows ports with USES=python to run default test (defined in bsd.port.mk) instead of python one. I did a similar one in perl5.mk. | |
| 612 | PYDISTUTILS_TESTARGS is undefined. Why not use TEST_ARGS? | |
Comment Actions
- Added NO_PYDISTUTILS_TEST like rP398474. To avoid consistency mess a la Uses/perl5.mk the change is expanded to include other targets as well. The naming continues the tradition of confusing distutils and setuptools.
- Added description for PYDISTUTILS_TESTARGS
- PYDISTUTILS_TESTARGS default remains as is. setuptools cannot handle make/gmake arguments:
$ make test PYDISTUTILS_TESTARGS='${TEST_ARGS}' -C devel/py-apscheduler ===> Testing for py27-apscheduler-3.0.3 ===> py27-apscheduler-3.0.3 depends on package: py27-pytest>=2.5.1 - found ===> py27-apscheduler-3.0.3 depends on package: py27-mock>0 - found invalid command name 'DESTDIR=/usr/ports/devel/py-apscheduler/work/stage' *** Error code 1
Comment Actions
After checking a few hybrid ports and thinking more:
- Drop NO_PYDISTUTILS_BUILD and NO_PYDISTUTILS_INSTALL as they're unlikely to work. NO_PYDISTUTILS_CONFIGURE is already handled but in different way.
- Use !defined(TEST_TARGET) similar to !defined(HAS_CONFIGURE) as make-style tests have to be explicitly enabled similar to configure.
Comment Actions
- Sync commit message as NO_PYDISTUTILS_TEST is gone. No need to mention TEST_TARGET explicitly as it's already documented in bsd.port.mk
| Mk/Uses/python.mk | ||
|---|---|---|
| 610 | This is not needed as we have NO_TEST. | |
| Mk/Uses/python.mk | ||
|---|---|---|
| 610 | No, they are different. IIUC NO_TEST is used to bypass test framework entirely. A port requires perl would add USES=perl5 but it does not mean that it wants to run perl tests. Since we do not support multiple tests currently, we need to check NO_PERL5_TEST in order to skip do-test: from perl5.mk but run the do-test: from bsd.port.mk or other Uses/*.mk (e.g. USES=python). Please try ftp/curl with/without NO_PERL5_TEST. BTW, I think Jan's idea is much better. Checking TEST_TARGET should have same effect but avoids new knob like NO_foo_TEST. | |
| Mk/Uses/python.mk | ||
|---|---|---|
| 610 |
https://reviews.freebsd.org/D3830
No. It won't work e.g. with perl5 as perl5 tests require TEST_TARGET, and here it's basically the same thing, but python duplicates *_TARGET by PYDISTUTILS_*_TARGET. Which feels wrong by itself. | |
Comment Actions
As it currently is, LGTM. && !defined(TEST_TARGET) check looks a bit pointless though. It is only useful for cases where pydistutils are used, however there's also Makefile specifically for running tests. Are there any real cases like that?
| Mk/Uses/python.mk | ||
|---|---|---|
| 612 | As per original diff comment, I would much prefer to use TEST_* variables in python.mk and python ports. These test framework bits ought to have nothing to do (and dont) with distutils/setuptools. The setuptools test command is just one possible invocation command. This allows us to use the plethora of python testing frameworks, with granularity in setting/overriding defaults from bsd.port.mk and python.mk if necessary. For example:
Thus, I'd like to see, in Python ports, and all ports more generally that support test suites, use something like: TEST_ENV TEST_CMD TEST_TARGET TEST_ARGS If the defaults (make test, python setup.py test, perl whatever) don't happen to just work for any package, it's then trivial to override the appropriate variable(s) for that port, while staying entirely faithful to a generic test framework. | |
| Mk/Uses/python.mk | ||
|---|---|---|
| 612 | As I've noted, I don't really like TEST_CMD.
Trying to sticking everything into "TEST_ENV TEST_CMD TEST_TARGET TEST_ARGS" or similar pattern will lead to ugliness like TEST_CMD=${python_CMD}
TEST_TARGET=./run_tests
TEST_ARGS=which is both longer and less readable than a custom do-test. | |
Comment Actions
Obsolete after 535f310d337c.
| Mk/Uses/python.mk | ||
|---|---|---|
| 612 |
DO_MAKE_TEST already can do something DO_MAKE_TEST= nosetests TEST_TARGET= # empty DO_MAKE_TEST= tox TEST_TARGET= # empty DO_MAKE_TEST= py.test TEST_TARGET= # empty DO_MAKE_TEST= ${PYTHON_CMD} -m unittest discover TEST_TARGET= # empty DO_MAKE_TEST= ${MAKE_CMD} test # with USE_PYTHON=distutils TEST_TARGET= # empty Most of default ${MAKE_ENV} is garbage at runtime, anyway. | |