security/py-obfsproxy-tor: Fix runtime dependency error
ClosedPublic
Actions

Authored by loader on Dec 11 2017, 7:32 PM.

Details

Reviewers

koobs
cs

Commits

rP468406: MFH: r468181
rP468181: security/py-obfsproxy-tor: Fix runtime dependency error

Summary

devel/py-twistedCore was removed since r450958:
https://svnweb.freebsd.org/ports?view=revision&revision=450958

# /usr/local/bin/obfsproxy
Traceback (most recent call last):
  File "/usr/local/bin/obfsproxy", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3195, in <module>
    @_call_aside
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3179, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3208, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 681, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 999, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 885, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Twisted_Core' distribution was not found and is required by obfsproxy

Proposed commit log message:

security/py-obfsproxy-tor: Fix runtime dependency error

- Change runtime dependency entry point Twisted_Core back to Twisted 
- Update checksum for CHEESESHOP
- Add test target

Reviewed_by: cs, koobs
Approved by: cs (maintainer), koobs (ports)
Differential_Revision: D13452
MFH: 2018Q2

Test Plan

portlint: OK (looks fine.)
testport: OK (poudriere: 1200060, [armv7, aarch64], '', py27 tested)
unittest: OK (passed: 80, skipped: 0, failed: 0)

Diff Detail

Repository

rP FreeBSD ports repository

Lint

No Lint Coverage

Unit

No Test Coverage

Build Status

Buildable 16036
Build 16012: arc lint + arc unit

Event Timeline

loader created this revision.Dec 11 2017, 7:32 PM

Herald added a subscriber: mat. · View Herald TranscriptDec 11 2017, 7:32 PM

loader edited the summary of this revision. (Show Details)Dec 11 2017, 7:34 PM

mat added inline comments.Dec 12 2017, 12:03 PM

security/obfsproxy/Makefile
25 ↗	(On Diff #36475)	Could this work with Python 3 ?

loader added inline comments.Dec 12 2017, 5:56 PM

security/obfsproxy/Makefile
25 ↗	(On Diff #36475)	No, this port couldn't work with Python 3. This reminds me that I made a mistake in D11562, the RUN_DEPENDS security/py-pyptlib couldn't work with Python 3 either. Sorry ...

Fixed in r463377.

Fix runtime dependency error

Harbormaster completed remote builds in B16036: Diff 41222.Apr 7 2018, 5:06 AM

loader edited the summary of this revision. (Show Details)Apr 7 2018, 5:11 AM

loader edited the test plan for this revision. (Show Details)

loader added a reviewer: cs.

loader edited the summary of this revision. (Show Details)

loader retitled this revision from security/obfsproxy: Fix build to security/obfsproxy: Fix runtime dependency error.

loader retitled this revision from security/obfsproxy: Fix runtime dependency error to security/py-obfsproxy-tor: Fix runtime dependency error.Apr 7 2018, 5:17 AM

cs accepted this revision.Apr 22 2018, 8:00 PM

This revision is now accepted and ready to land.Apr 22 2018, 8:00 PM

LGTM, commit should be merged to quarterly if quarterly version of the port is also broken (MFH: 2018Q2)

Also: I would also check that argparse isn't actually required. Just removing it from setup.py stops a runtime error from occurring at startup, but doesn't prevent functionality loss if the code actually references it, or requires the third party package.

Just because argparse is included in the Python stdlib does not by itself mean that python packages can/do use the stdlib version. They are not the same thing, and removal of argparse from dependencies has caused broken ports in the past.

See Also, among others: http://svnweb.freebsd.org/changeset/ports/440641

Hi koobs@, thanks for the note, I wasn't aware that the newer version of
devel/py-argarse might be required. The argparse module from stdlib works
for security/py-obfsproxy-tor, should I change it back to use devel/py-argparse?

% obfsproxy -v
0.2.13

% obfsproxy -h
usage: obfsproxy [-h] [-v] [--log-file LOG_FILE]
                 [--log-min-severity {error,warning,info,debug}] [--no-log]
                 [--no-safe-logging] [--data-dir DATA_DIR] [--proxy PROXY]
                 {managed,obfs2,dummy,obfs3,scramblesuit,b64} ...

py-obfsproxy: A pluggable transports proxy written in Python

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  --log-file LOG_FILE   set logfile
  --log-min-severity {error,warning,info,debug}
                        set minimum logging severity (default: None)
  --no-log              disable logging
  --no-safe-logging     disable safe (scrubbed address) logging
  --data-dir DATA_DIR   where persistent information should be stored.
  --proxy PROXY         Outgoing proxy (<proxy_type>://[<user_name>][:<passwor
                        d>][@]<ip>:<port>)

supported transports:
  {managed,obfs2,dummy,obfs3,scramblesuit,b64}
    managed             managed mode
    obfs2               obfs2 help
    dummy               dummy help
    obfs3               obfs3 help
    scramblesuit        scramblesuit help
    b64                 b64 help

Another question is how can we make sure Python scripts load
the argparse module from PYTHON_SITELIBDIR instead of PYTHON_LIBDIR.

There are five ports which have devel/py-argparse in RUN_DEPENDS:

converters/py-zfec
devel/py-dateutils
devel/youcompleteme
security/py-gixy
textproc/pootle

I just checked converters/py-zfec, devel/py-dateutils and security/py-gixy,
it seems they just load the python2.7/argparse.py{,c} and doesn't check for
the python2.7/site-packages/argparse.py{,c}:

py27-zfec-1.5.2:

% truss /usr/local/bin/zfec 2>&1 | grep '\/argparse\.'
open("/usr/local/lib/python2.7/site-packages/zfec/argparse.so",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/site-packages/zfec/argparse.py",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/site-packages/zfec/argparse.pyc",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/bin/argparse.so",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/bin/argparse.py",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/bin/argparse.pyc",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/argparse.so",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/argparse.py",O_RDONLY,0666) = 5 (0x5)
open("/usr/local/lib/python2.7/argparse.pyc",O_RDONLY,0666) = 6 (0x6)

py27-dateutils-0.6.6:

% truss /usr/local/bin/datediff 2>&1 | grep '\/argparse\.'
open("/usr/local/lib/python2.7/site-packages/dateutils/argparse.so",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/site-packages/dateutils/argparse.py",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/site-packages/dateutils/argparse.pyc",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/bin/argparse.so",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/bin/argparse.py",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/bin/argparse.pyc",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/argparse.so",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/argparse.py",O_RDONLY,0666) = 4 (0x4)
open("/usr/local/lib/python2.7/argparse.pyc",O_RDONLY,0666) = 5 (0x5)

py27-gixy-0.1.3_1:

% truss /usr/local/bin/gixy 2>&1 | grep '\/argparse\.'
open("/usr/local/bin/argparse.so",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/bin/argparse.py",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/bin/argparse.pyc",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/argparse.so",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/python2.7/argparse.py",O_RDONLY,0666) = 6 (0x6)
open("/usr/local/lib/python2.7/argparse.pyc",O_RDONLY,0666) = 7 (0x7)

In D13452#319545, @loader wrote:

Hi koobs@, thanks for the note, I wasn't aware that the newer version of
devel/py-argarse might be required. The argparse module from stdlib works
for security/py-obfsproxy-tor, should I change it back to use devel/py-argparse?

If it indeed works without it in this case (ie: the install_requires is actually spurious), then its fine to stay as is. However: some code may have try/fallback code to use the package if it exists, falling back to the stdlib if it doesn't, and/or vice versa.

If packages have spurious (unnecessary) argparse, or any other package requirements set (but don't use it), sure they are OK to be removed (and patched out), BUT its better removed upstream.

The bottom line is we shouldn't be doing anything in our ports which deviates from, or beyond what the upstream package says it wants/needs, because if we do, we *must* take on the *full* burden of doing so.

If the metadata (like requirements) isn't reflected in actual code (use), then upstream changes should be submitted, with us carrying a local ports patch (if absolutely necessary), until the changes are released. In most if not all cases, the benefit does not come close to outweighing that extra burden, vs just submitting changes upstream.

Finally, every time I've seen it happen in freebsd ports, it has been wrong and broken things in non-obvious ways.

Thank you @koobs for your help and guidance. :)

loader edited the summary of this revision. (Show Details)Apr 24 2018, 6:18 AM

Closed by commit rP468181: security/py-obfsproxy-tor: Fix runtime dependency error (authored by loader). · Explain WhyApr 24 2018, 6:31 AM

This revision was automatically updated to reflect the committed changes.