add shared py-backports namespace file for backports.* modules

[lifanov]

Summary

All backports.* python modules install the same file.
The proposal is to remove this file from the packages
and install it separately. This is the first prototype.

Step 2 could be one of:
o provide USES=python:backports, which will automatically remove this file from plist and add dependency on devel/py-backports (and maybe additionally imply python:2)
o make USES=python react to PORTNAME and do the right thing
o patch all py-backports.* files to remove this file

I like the first approach above

See Also:

• https://www.python.org/dev/peps/pep-0420/
• https://github.com/pypa/setuptools/issues/960
• https://github.com/pypa/setuptools/issues/406
• https://github.com/pypa/python-packaging-user-guide/issues/265

[mat]

devel/py-backports/Makefile

19–21

Never use %%foo%% placeholders in PLIST_FILES, always use the variable it comes from, so, here, ${PYTHON_SITELIBDIR} and ${PYTHON_PYOEXTENSION}.

devel/py-backports/pkg-descr

2

pkg-descr must, must, be longer than COMMENT, at least a couple of lines.

[lifanov]

address mat's feedback:

o use variables in PLIST_FILES directly
o provide package description that is more descriptive than the comment

[mat]

devel/py-backports/Makefile

24

It just a thought, but this will probably not do what you think:

$ echo '__path__ = __import__('pkgutil').extend_path(__path__, __name__)'
__path__ = __import__(pkgutil).extend_path(__path__, __name__)

Notice the lack of quotes. Instead of playing around with quotes, it might be better to drop the file in the files directory and INSTALL_DATA it.

Also, never, ever, use ${ECHO} in the ports tree. From Mk/bsd.commands.mk:

# ECHO is defined in /usr/share/mk/sys.mk, which can either be "echo",
# or "true" if the make flag -s is given.  Use ECHO_CMD where you mean
# the echo command.
ECHO_CMD?=              echo    # Shell builtin

# Used to print all the '===>' style prompts - override this to turn them off.
ECHO_MSG?=              ${ECHO_CMD}

[lifanov]

address mat's feedback:

o use ${ECHO_CMD} instead of ${ECHO}, which is defined in mk/sys.mk
o fix quoting around 'pkgutil'

I would like to not create a separate file in files/ for just one line
of content, but I don't feel strongly about it either.

[lifanov]

whoops, included too much

[yuri_rawbw.com]

What is holding this?

[yuri_rawbw.com]

Here is an idea for a different approach.

Have a package manager silently ignore file conflicts when files are identical. pkg can keep track of how many times that file is installed, and remove it after the last package that installed it is removed. This can be implemented with a simple reference count.

Then py-backports-* can just make sure the contents are the same.

Pros: 1. Solves this py-backports-* problem without creating extra packages, 2. might be useful in some other situations.
Cons: Looks like a far-fetched, fancy feature.

[koobs]

The absence of something holding a change/proposal is necessary but insufficient (alone) to promote a change.

In this case, sufficient review and investigation of potential alternatives/improvements

In D11095#242627, @yuri_rawbw.com wrote:

Here is an idea for a different approach.

Have a package manager silently ignore file conflicts when files are identical. pkg can keep track of how many times that file is installed, and remove it after the last package that installed it is removed. This can be implemented with a simple reference count.

Then py-backports-* can just make sure the contents are the same.

Pros: 1. Solves this py-backports-* problem without creating extra packages, 2. might be useful in some other situations.
Cons: Looks like a far-fetched, fancy feature.

-1 on pkg being the place to solve this. Though there may be merit in improved conflicts handling (in certain limited cases) in pkg in general, I'd like not to have the path of least resistance for porters/maintainers be let a tool handle an issue rather than do it properly (where properly means ports should not conflict in the first place)

@lifanov Could you add an example consumer port to this changeset please

[yuri_rawbw.com]

In D11095#242930, @koobs wrote:

-1 on pkg being the place to solve this. Though there may be merit in improved conflicts handling (in certain limited cases) in pkg in general, I'd like not to have the path of least resistance for porters/maintainers be let a tool handle an issue rather than do it properly (where properly means ports should not conflict in the first place)

Least resistance wasn't my motivation. Adding a new pkg feature is actually many times more work, and longer wait. I really believe that this feature is reasonable to have in pkg, and can be useful in other cases as well.

[koobs]

In D11095#242931, @yuri_rawbw.com wrote:

In D11095#242930, @koobs wrote:

-1 on pkg being the place to solve this. Though there may be merit in improved conflicts handling (in certain limited cases) in pkg in general, I'd like not to have the path of least resistance for porters/maintainers be let a tool handle an issue rather than do it properly (where properly means ports should not conflict in the first place)

Least resistance wasn't my motivation. Adding a new pkg feature is actually many times more work, and longer wait. I really believe that this feature is reasonable to have in pkg, and can be useful in other cases as well.

I meant as a potential downside symptom, not that that was an intent or desirable property of the proposal

[antoine]

devel/py-backports/Makefile

27

The correct way to compileall is:

${PYTHON_CMD} -m compileall -d ${PYTHON_SITELIBDIR} ${STAGEDIR}${PYTHON_SITELIBDIR}
${PYTHON_CMD} -O -m compileall -d ${PYTHON_SITELIBDIR} ${STAGEDIR}${PYTHON_SITELIBDIR}

[antoine]

devel/py-backports/Makefile

27

And yes, more than half of the ports do it wrong... sigh...

[lifanov]

o use compileall with -d flag
o add a sample consumer to the patch

[antoine]

devel/py-backports.csv/Makefile

23 ↗

(On Diff #32443)

Removing it from stagedir is not enough, you have to remove it from plist too.

[lifanov]

I would like some help with filtering files out of generated plist. I can't figure out how to do this. I tested conversion of a port that actually needs this and it *is* necessary.

[antoine]

devel/py-backports.csv/Makefile

23 ↗

(On Diff #32443)

Note that backports.csv is a bad example, it doesn't install any __init__.py

[lifanov]

Yep. I tried it with security/py-backports.ssl_match_hostname and... it doesn't work.
I'll switch the test port to that one once I figure out a correct fix.

I *could* just sed it out of egginfo post-extract. Is this a good idea?

[antoine]

In D11095#252326, @lifanov wrote:

I would like some help with filtering files out of generated plist. I can't figure out how to do this. I tested conversion of a port that actually needs this and it *is* necessary.

You may use something like:

POST_PLIST= remove-init-from-plist
remove-init-from-plist:

${REINPLACE_CMD} '/backports\/__init__.py/d' ${TMPPLIST}

Note that with python 3 some files are installed differently, in backports/__pycache__/__init__.*

[lifanov]

Cool, thanks! The backports.* stuff is backports of Python 3 features to Python 2, so these should not be necessary for Python 3 at all.

[antoine]

In D11095#252330, @lifanov wrote:

Cool, thanks! The backports.* stuff is backports of Python 3 features to Python 2, so these should not be necessary for Python 3 at all.

No, for instance backports.ssl_match_hostname is a backport from python3.5 so it is necessary on python3.3 and python3.4

[lifanov]

Ahh, you're right!

[antoine]

FYI openbsd ports solved the problem by adding py-backports too : http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/devel/py-backports/

[lifanov]

Oooh, there is a Gentoo mirror for this package. Hold on while I shamelessly rip off the OpenBSD port...

[lifanov]

fix stage and plist issues using a post-stage and post-plist target

[lifanov]

really fix for and test with Python 3

[lifanov]

At this point, the patch works and I can commit this port and security/py-backports.ssl_match_hostname if they are ready.

[leres]

Are we going to get to the point when it's possible to have devel/pylint and devel/ipython installed at the same time again?

[antoine]

Is there some progress on this issue?

[antoine]

With the current ports tree I have the problem with backports.shutil_get_terminal_size vs backports.functools_lru_cache:

pkg-static: py27-backports.shutil_get_terminal_size-1.0.0 conflicts with py27-backports.functools_lru_cache-1.4 (installs files into the same place). Problematic file: /usr/local/lib/python2.7/site-packages/backports/__init__.py

[antoine]

I updated your patch and put it on https://people.freebsd.org/~antoine/ports/backports.diff

[lifanov]

Thanks Antoine!

I was waiting for an approval from someone from Python.
I will re-test your patch and update this revision.

[lifanov]

update patch with the one fixed by antoine

This includes conversion of two more ports:
devel/py-backports.functools_lru_cache
devel/py-backports.shutil_get_terminal_size

I tested it and all ports work with all flavors.

[lifanov]

Your patch works, thank you!
This is pending approval by Python.

[antoine]

You can commit approved by portmgr.

[lifanov]

Thank you!