Page MenuHomeFreeBSD

Mk/Uses/python.mk: add support for PyPy
AbandonedPublic

Authored by dbn on Sep 29 2016, 6:32 AM.

Details

Reviewers
antoine
koobs
mat
Group Reviewers
portmgr
Python
Summary

Add support for PyPy to USES+=python. This adds support for:

  1. specifying a port requires pypy (or cpython) specifically, and
  2. allows the user to specify a DEFAULTS=python=pypy2.7

The patch works as follows:
i) python versions refer to the language version, not the implementation version (for cpython it is the same thing, for pypy it differs)
ii) DEFAULTS and PYTHON_VERSION now require the implementation name (e.g. cpython2.7 or pypy2.7). python is an alias for cpython.

Todo:
e) Investigate all uses of PYTHON_VERSION in Ports (24 references remain)

Done:
a) add relevant pypy27-setuptools and pypy33-setuptools ports
c) add tests for DEFAULTS vs PYTHON_VERSION detection (a la lang/python, lang/python2 and lang/python3)
d) change default back to cpython
g) restrict PYTHON_VERSION to valid values only
h) check Python implementation is consistent

Follow up changes:
b) automatically map py-$PORTNAME to their pypy-$PORTNAME equivalents, if it exists
f) add pypy33 slave ports for pypy-(sqlite|gdbm|tkinter)
j) exp-run (with pypy as default)

Test Plan

i) http://people.freebsd.org/~dbn/testMkPython.py (updated to 1.3.1)
ii) exp-run (PR 213737)

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
dbn set the repository for this revision to rP FreeBSD ports repository.
dbn edited edge metadata.

Couple of questions.

  • What do USES=python.pypy or USES=python.cpython do exactly ?
  • What happens to people who have DEFAULT_VERSIONS+=python=2.7 ?

Also, could you use arcanist to upload the diff, or if you are allergic to it, could you generate the diff like it does so we get context with svn diff --diff-cmd=diff -x -U9999 or if using git, with git diff -U9999.

dbn updated this object.
dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.
  1. fix: PYTHON_VERSION not set correctly
  2. fix: PLIST_SUB used old definition of PYTHON_VERSION
  3. fix: prefer PYTHON_DEFAULT over PYTHON2_DEFAULT
  4. update: added slave ports for setuptools for pypy
  5. update: tests at version 0.2.0 (test for (1) and (2))
In D8069#166958, @mat wrote:

Couple of questions.

  • What do USES=python.pypy or USES=python.cpython do exactly ?

The above says: this port only works with python intepreter X. I.E. what USES=python:2.7 says about version USES=python:pypy says about implementation.

  • What happens to people who have DEFAULT_VERSIONS+=python=2.7 ?

python.mk defaults to the cpython implementation when no implementation is specified (and adds a warning). I.E> the above becomes DEFAULT_VERSIONS+=python=cpython2.7 (see line 275 of the patched python.mk).

Also, could you use arcanist to upload the diff, or if you are allergic to it, could you generate the diff like it does so we get context with svn diff --diff-cmd=diff -x -U9999 or if using git, with git diff -U9999.

Currently the patch sits in svn (with phabricator acting as the revision control). I'll give learning arcanist a try (but for now the svn command worked).

In D8069#168194, @dbn wrote:
In D8069#166958, @mat wrote:

Couple of questions.

  • What do USES=python.pypy or USES=python.cpython do exactly ?

The above says: this port only works with python intepreter X. I.E. what USES=python:2.7 says about version USES=python:pypy says about implementation.

Sure, but I am talking about USES=python.pypy and USES=python.cpython, not USES=python:pypy or USES=python:cpython.

dbn updated this object.
dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.
  1. fix: add pypyX-setuptools to Makefile's SUBDIR
  2. fix: merge python.pypy.mk and python.cpython.mk into python.mk
  3. feature: add checks to ensure supported version of Python (g)
  4. feature: add consistency checks for Python implementation
  5. test: update tests to 0.3.0
In D8069#168321, @mat wrote:
In D8069#168194, @dbn wrote:
In D8069#166958, @mat wrote:

Couple of questions.

  • What do USES=python.pypy or USES=python.cpython do exactly ?

The above says: this port only works with python intepreter X. I.E. what USES=python:2.7 says about version USES=python:pypy says about implementation.

Sure, but I am talking about USES=python.pypy and USES=python.cpython, not USES=python:pypy or USES=python:cpython.

Sorry, I completely misunderstood what you were asking. The reason is also the answer: python.pypy.mk and python.cpython.mk are supposed to be private. I have updated the patch by merging these files into python.mk.

Mk/bsd.default-versions.mk
73–79

Shouldn't the default be kept to lang/python* ?

dbn updated this object.
dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.
  1. fix: _PYTHON_DEFAULT_VERSION from $LOCALBASE/bin/python (test updated)
  2. fix: indentation of some blocks in python.mk
  3. update: use of PYTHON_VERSION in Mk/Uses/(kde|zope).mk
  4. update: changed default back to cpython
  5. update: setting of PYTHON_VERSION in pyXY-${PORTNAME} ports
  6. update: some uses of PYTHON_VERSION by ports
dbn marked an inline comment as done.Oct 6 2016, 6:31 AM
Mk/Uses/kde.mk
164–165

This will not work, USES entries are most often sorted alphabetically, and this means python.mk will be parsed after kde.mk, and PYTHON_SUFFIX will not be defined at that point, whereas PYTHON_VERSION is guaranteed to be.

dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.
  1. fix: kde's use of PYTHON_VERSION
  2. fix: IGNORE warnings
  3. fix: add missing autotranslations from (python)?X.Y -> cpythonX.Y
dbn marked an inline comment as done.Oct 6 2016, 3:28 PM
dbn added inline comments.
Mk/Uses/kde.mk
164–165

Good catch, however PYTHON_VERSION is not guaranteed to be available. It is only available if:
a) the user specifies PYTHON_VERSION, or
b) the port specifies PYTHON_VERSION, or
c) python.mk has been parsed/included, or
d) it is being built as a dependency of a port that USES=python.

I included ${USESDIR}/python.mk, if needed, to get the correct version of PYTHON_VERSION.

Mk/Uses/kde.mk
164–165

Ah, sorry, I meant PYTHON_DEFAULT, not PYTHON_VERSION...

I really don't like all that spaghetti code with many uses includ'ing each other.

As the only thing this is used for is to add stuff to PLIST_SUB, maybe it could go in a _POST section here, and you would not need to include it. See kmod.mk or autoconf.mk for what I mean about a _POST section.

dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.
dbn marked an inline comment as done.
  • fix: eliminate including python.mk in kde.mk
dbn marked 2 inline comments as done.Oct 10 2016, 5:39 AM
dbn added inline comments.
Mk/Uses/kde.mk
164–165

I didn't know about the post section. I'm sure it is very useful in getting USES to interact cleanly with each other.

I have updated kde per your suggestion, and updated the test to cover both the Python 3 and Python 2 case.

dbn marked 2 inline comments as done.Oct 10 2016, 5:40 AM

All the use of %%PYTHON_VERSION%% or ${PYTHON_VERSION} in the ports tree have to be checked too.

Mk/bsd.default-versions.mk
77

should be cpython3.4

In D8069#170258, @dbn wrote:
  • fix: eliminate including python.mk in kde.mk

Thank you :-)

Maybe it is mature enough to ask for an exp-run ?

dbn updated this object.
dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.
  • fix: remove PYTHON_VERSION from autofix list (unbreak lang/python[23])
  • fix: remove extra '}' from _DEPENDS arguments
  • fix: checking _PYTHON_VERSION againsted supported versions
  • test: check _DEPENDS

(I hope this works, I now have a joint git/svn tracking thing going)

dbn marked an inline comment as done.Oct 13 2016, 7:29 PM

All the use of %%PYTHON_VERSION%% or ${PYTHON_VERSION} in the ports tree have to be checked too.

This is an in-progress item (see item (e)).

In D8069#170825, @mat wrote:

Maybe it is mature enough to ask for an exp-run ?

I tried a mini exp-run at home and got the following results:

Queued Built Failed Skipped Ignored Remaining
1066 484 18 560 4 0

Where 2 of the skipped ports were python2 and python3... But after a few fixes (per this update) we now get:

Queued Built Failed Skipped Ignored Remaining
582 284 16 277 5 0

with libclc blocking 269 ports. Progesss slowly happening.

dbn marked an inline comment as not done.Oct 13 2016, 7:30 PM
In D8069#171289, @dbn wrote:

Queued Built Failed Skipped Ignored Remaining
582 284 16 277 5 0

Progress is slowly happening! The current status is:
Failed Skipped Ignored
23 10 6

There are 76 remaining uses of PYTHON_VERSION to be checked.

dbn updated this object.
dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.

devel/libclc: replace ${PYTHON_VERSION} with ${PYTHON_CMD}

USES=python: fix concurrent's default link.

  • Fix the comparison of ${PYTHON_DEFAULT_VERSION} against ${_PYTHON_VERSION}. The latter needs the ${_PYTHON_IMPL} as prefix.

devel/libsoup: use USES=shebangfix instead of ${PYTHON_VERSION}

devel/py-gobject: pkg-plist doesn't use implementation specific name.

www/webkit-qt5: replace ${PYTHON_VERSION} with ${PYTHON_CMD}

  • Remove 'python' from the PATH checks as we use the absolute path.

devel/libsearpc: doesn't use implementation specific install directory.

USES=python: update documentation for changes with PYTHON_VERSION.

USES=python: add PYTHON_LIB variable.

  • Some ports need to know which library to link to.

devel/py-cxx: installs files in an implementation agnostic directory.

net-mgmt/ccnet: doesn't use implementation specific install directory.

sysutils/py-zfs: link against PYTHON_LIB instead of PYTHON_VERSION.

devel/newt: fix build with python

  • use PYTHON_CMD instead of PYTHON_VERSION
  • install python files into PYTHON_SITELIBDIR instead of manually constructed path (in Makefile.in)
  • fix build location for python files (only a single version of python is supported, so no need for different directories per python version)

devel/pysvn: use PYTHON_VER instead of truncating PYTHON_VERSION

print/system-config-printer: remove redundant PYTHON_VER from PLIST_SUB

sysutils/condor: use PYTHON_INCLUDEDIR and PYTHON_LIB instead

  • Switch from PYTHON_LIBRARY:FILEPATH to PYTHON_LIBRARIES. The later correctly links against the python library.

databases/metakit: specify python's include and sitelib via configure.

  • the --with-python accepts the include and sitelib paths as a comma separated argument.
  • modernise port with option helpers.

devel/py-ice: use PYTHON_LIB instead of PYTHON_VERSION.

  • Define PYTHON(=PYTHON_CMD) in MAKE_ENV so that py-ice can detect the correct include headers.
  • Override the library flags to point directly to PYTHON_LIB.

games/kiki: use correct PYTHON_ variables.

  • Change patch to use PYTHON_INCLUEDIR instead of assumed path
  • Change patch to use PYTHON_LIBDIR instead of assumed path
  • Use PYTHON_VER instead of inferring it from PYTHON_VERSION
  • Change patch to link directly against PYTHON_LIB

devel/py-ocempgui: use PYTHON_INCLUEDIR instead of inferring it from PYTHON_VERSION

mail/opensmtpd-extras-*-python: fix detection of python-config.

  • Specify the prefix in which the python command can be found
  • Get python command name from PYTHON_CMD instead of assuming it is PYTHON_VERSION

net-mgmt/seafile: fix use of PYTHON_VERSION

  • use PYTHON_CMD where python is being invoked
  • use PYTHON_VER instead of a implementation aware PYTHON_VERSION

graphics/py-ming: link directly to PYTHON_LIB

games/pykawari: link directly against PYTHON_LIB

math/qtiplot: link directly against PYTHON_LIB

cad/freecad: use implementation agnostic version

editors/libreoffice(4): use PYTHON_CMD instead of PYTHON_VERSION

www/uzbl: use PYTHON_CMD instead of PYTHON_VERSION

devel/gps: use PYTHON_LIBDIR instead of PYTHON_VERSION

USES=python: add PYTHON_ABIVER to PYTHON_LIB for cpython.

TL;DR; The patch is maturing and "almost" ready:

  1. On my local mini exp-run I managed to get all ports to compile.
  2. I have requested an full exp-run to test the patch
  3. There are still 60 uses of PYTHON_VERSION that haven't been vetted (although those ports do not break)
Mk/Uses/python.mk
104

Would it not be better to rename this to PYTHON_LDFLAGS and define it as: -L${LOCALBASE}/lib -lpython${PYTHON_VER}${PYTHON_ABIVER}

The existing behaviour does work but not sure how correct it is. It will be quite easy to convert all references of PYTHON_LIB to PYTHON_LDFLAGS.

dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.

USES=python: fixes for fmake.

  • fmake cannot compare against a constant (or for loop variable) on the left hand side.
  • fmake cannot compare against two variables on the left hand side.

Time has run out for me. I'm AFK until 21 November. I'll resume this work then (unless anyone beats me to it ;-)).

dbn updated this object.
dbn edited edge metadata.
  • feature: add 'pydepends' to add package building guard (i.e. IGNORE port if it's dependencies are not compatible with PYTHON_DEFAULT). This is auto-detected.
  • fix: many uses of PYTHON_VERSION and other variables.

I don't think that the conditional
+.if ${BUILD_DEPENDS:M*\:*/py-*} || ${RUN_DEPENDS:M*\:*/py-*}
for the 'pydepends' feature will work correctly (packages that are currently built will no longer be built)

for instance devel/py-asyncio or devel/py-setuptools35 build versions for python 3.5 and the ports depending on them are built right now on the package builders.

dbn edited edge metadata.

Update the package building guard, remove keyword.

Unfortunately this slows down ports that USES=python due to the requirement to accurately check the dependencies (it spawns a sub-make process).

I think, strongly, that having an automatic mechanism to add the package guard is useful (until poudriere gains the ability for multiple packages from a port). For example, if we switch to python3 as the default (or even if a user does this) then the automatic machanism will work nicely.

However, I don't want this change to delay this change and will happily:
a) drop the autodetection and go with 'py3depends' (note the name change), or
b) just fix the guards, in place.
(or c) accept, or quickly fix the existing auto-detect guard).

Which route should we take?

  • For the package building guard, I prefer a) or b) as it is more conservative for now and annotates ports for which dependencies will have to be "flavored" in priority
  • The audio/libofa changes seem unrelated
  • In www/webkit-qt5/Makefile, the "${REINPLACE_CMD} -e 's,python,,g'" looks strange, changing the python to ${PYTHON_CMD:C/.*\///} would be less strange

Also could 1 or 2 people from python@ review this before the exp-run? it's better if there is some consensus on the USE_PYTHON= and DEFAULT_VERSIONS=python= changes

Mk/bsd.default-versions.mk
73

missing a comma at the end.

77

I don't think pypy3.3 is possible here.

Mk/bsd.default-versions.mk
77

Mmmm, sorry, pypy2.7 is the one that should not be here.

This review is now too old, and way behind all the changes that happened to USES=python (especially the introduction of FLAVORS).