Page MenuHomeFreeBSD

Add Q/A tests and shebang fixes for python[23] shebangs
ClosedPublic

Authored by AMDmi3 on Dec 21 2017, 1:09 PM.

Details

Summary

This is kind of followup to D9332: we've added checks and fixes for
python, but not for python2/3 interpreters. Currently there's an ongoing
effort to switch instances of e.g. USES=python:2 to USES=python:2.7,
which makes ports no longer depend on python2 metaport and thus allows
a situation where python2/python3 executable is not available. This
can be tested with a simple test port:

PORTNAME=       testport
PORTVERSION=    0.0.0
CATEGORIES=     sysutils
DISTFILES=      #

MAINTAINER=     amdmi3@FreeBSD.org
COMMENT=        #

USES=           python
SHEBANG_FILES=  *.py
NO_BUILD=       yes

PLIST_FILES=    bin/python.py \
                bin/env_python.py \
                bin/python2.py \
                bin/env_python2.py \
                bin/python3.py \
                bin/env_python3.py \
                bin/python_cmd.py

OPTIONS_DEFINE= SHEBANGFIX
SHEBANGFIX_USES=shebangfix

do-extract:
        @${MKDIR} ${WRKSRC}
        echo '#!/usr/local/bin/python' > ${WRKSRC}/python.py
        echo '#!/usr/bin/env python' > ${WRKSRC}/env_python.py
        echo '#!/usr/local/bin/python2' > ${WRKSRC}/python2.py
        echo '#!/usr/bin/env python2' > ${WRKSRC}/env_python2.py
        echo '#!/usr/local/bin/python3' > ${WRKSRC}/python3.py
        echo '#!/usr/bin/env python3' > ${WRKSRC}/env_python3.py
        echo '#!${python_CMD}' > ${WRKSRC}/python_cmd.py

do-install:
        ${INSTALL_SCRIPT} ${WRKSRC}/*.py ${STAGEDIR}${PREFIX}/bin
.for f in ${PLIST_FILES}
        @if ${STAGEDIR}${PREFIX}/${f}; then \
                echo ${f}: OK; \
        else \
                echo ${f}: Failed; \
        fi 2>/dev/null
.endfor

It installs python scripts with different shebangs so one could
examing stage Q/A output and the actual ability to run scripts.

So, currently the output is the following:

SHEBANG=off
bin/python.py: Failed
bin/env_python.py: Failed
bin/python2.py: Failed
bin/env_python2.py: Failed
bin/python3.py: Failed
bin/env_python3.py: Failed
bin/python_cmd.py: OK
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/python.py'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/env_python.py'
SHEBANG=on
bin/python.py: OK
bin/env_python.py: OK
bin/python2.py: Failed
bin/env_python2.py: Failed
bin/python3.py: Failed
bin/env_python3.py: Failed
bin/python_cmd.py: OK

As you can see, only scripts with /usr/local/bin/python and
/usr/bin/env python interpreters were detected as incorrect
shebangs and fixed by shebangfix. python2/pytnon3 are neither
detected neither runnable.

So I propose to add corresponding checks and fixes.

Test Plan

After the patch applied,

SHEBANG=off
Error: '/usr/bin/env python3' is an invalid shebang you need USES=shebangfix for 'bin/env_python3.py'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/python.py'
Error: '/usr/local/bin/python2' is an invalid shebang you need USES=shebangfix for 'bin/python2.py'
Error: '/usr/local/bin/python3' is an invalid shebang you need USES=shebangfix for 'bin/python3.py'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/env_python.py'
Error: '/usr/bin/env python2' is an invalid shebang you need USES=shebangfix for 'bin/env_python2.py'
SHEBANG=on
bin/python.py: OK
bin/env_python.py: OK
bin/python2.py: OK
bin/env_python2.py: OK
bin/python3.py: OK
bin/env_python3.py: OK
bin/python_cmd.py: OK

That is, all incorrect shebangs are either detected by stage-qa or fixed by shebangfix.

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

I had a thing similar to that, but please go ahead.

This revision is now accepted and ready to land.Dec 21 2017, 1:19 PM
This revision was automatically updated to reflect the committed changes.

Thanks! Out of curiocity, was the possibility of automatically adding dependency on python2/3 metaports considered?

Thanks! Out of curiosity, was the possibility of automatically adding dependency on python2/3 metaports considered?

How would one guess that one needs the metaport, and what good would it achieve ?

How would one guess that one needs the metaport, and what good would it achieve ?

I guess it makes sense to always add a metaport for corresponding major python version (2 or 3), and it will save us from caring of shebangs, at least partially.
There's an issue of multiple minor versions (3.4, 3.5, 3.6) this won't fix though...

Those two metaports never really made sense, they were transitional to help while we cleaned the tree, they should probably be removed, or really made as meta ports, empty shells with only USES=python:${PYTHONX_DEFAULT}.

In D13571#283859, @mat wrote:

Those two metaports never really made sense, they were transitional to help while we cleaned the tree, they should probably be removed, or really made as meta ports, empty shells with only USES=python:${PYTHONX_DEFAULT}.

There is sense in having python/python2/python3 links as that's what most people interact with and use in both command line and shebangs.

There is sense in having python/python2/python3 links as that's what most people interact with and use in both command line and shebangs.

We have that ability now in USES=uniquefiles. Ruby does it well, and python could leverage it too. At build-time, python knows whether it's the default python and/or default python2/3.