Page MenuHomeFreeBSD

lang/pypy: simplify port
ClosedPublic

Authored by dbn on Jul 26 2015, 8:21 PM.

Details

Summary
  • Use upstreams directory layout instead of FreeBSD's preferred directory layout [1]
  • Move the pypy installation into $PREFIX/pypy-x.y
  • Remove the ability to build multiple instances (i.e. default to just the JIT instance)
  • Install the binary as pypy (instead of pypy-2.6 with a symlink to pypy)
  • Remove the creation of cffi modules (will be outsourced to other ports, a la cpython)
  • Remove sqlite3 and gdbm dependencies (i.e. cffi modules)

PR: 183795

Test Plan
  • Run the patch through poudriere for a few permutations.
  • Run the regression-test target (currently blocking on hard-coded reference to g++)
  • Bootstap the port [DONE]
  • portlint [DONE]

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

dbn updated this revision to Diff 7348.Jul 26 2015, 8:21 PM
dbn retitled this revision from to lang/pypy: simplify port.
dbn updated this object.
dbn edited the test plan for this revision. (Show Details)
dbn added reviewers: koobs, alfred, robak.
dbn set the repository for this revision to rP FreeBSD ports repository.
dbn added a project: Python.
dbn added a subscriber: Python.
dbn updated this revision to Diff 7349.Jul 26 2015, 8:24 PM

Fix OPTIONS as identified by portlint.

robak edited edge metadata.Jul 26 2015, 8:30 PM

I'll run this through my full 8/9/10/11 i386/amd64 poudriere test suite and will let you know my results. Thanks for fixing that!

robak added a comment.Jul 26 2015, 8:51 PM

The patch fails to apply cleanly:

Patching file files/patch-rpython__rtyper__tool__rffi_platform.py using Plan A...
Hunk #1 succeeded at 0.
Removing files/patch-rpython__rtyper__tool__rffi_platform.py (empty after patching).
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: files/patch-rpython_config_support.py
|===================================================================
|--- files/patch-rpython_config_support.py
|+++ files/patch-rpython_config_support.py
--------------------------
File to patch: files/patch-rpython_config_support.py
No file found--skip this patch? [n]
patch: **** can't find files/patch-rpython_config_support.py
dbn updated this revision to Diff 7359.Jul 27 2015, 5:08 AM
dbn updated this object.
dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.
  • Update patch using svn diff --patch-compatible.
  • Fix OPTIONS to actually show option under correct situations
  • Remove '`' around ${WRKDIR}/build.

While failing to build on 8, it does build fine on 10.1 amd64. I am testing the build for 10.1 i386 now, and if that will work, I'll move to testing on 9, and then I'll do tests with virtualenv.

lwhsu added a subscriber: lwhsu.Jul 27 2015, 11:03 AM
In D3209#64602, @robak wrote:

While failing to build on 8, it does build fine on 10.1 amd64. I am testing the build for 10.1 i386 now, and if that will work, I'll move to testing on 9, and then I'll do tests with virtualenv.

I think you can just ignore 8, its support cycle is less than a week left.

dbn updated this revision to Diff 7396.Jul 27 2015, 7:46 PM

@lwhsu: while I agree FreeBSD 8 can be ignored, this is a bug in the code.

I failed to pass in $MAKE_ENV in the translation step - thus the clang base compiler was being used instead of of $CC. This will work where clang is available (and sufficient) but makes USES=compiler:c11 redundant and fails to honor user requests.

This patch fixes this but should not require a full retest.

dbn added a comment.EditedJul 27 2015, 7:54 PM

I have encountered a strange problem in my testing:

# make install
<snip/>
# pypy
Shared object "libpypy-c.so" not found, required by "pypy"
# cd /usr/local/bin
# ./pypy
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 26 2015, 18:38:23)
[PyPy 2.6.0 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.

The linking error happens irrespective of work directory (i.e. $LOCALBASE/bin or $LOCALBASE/pypy-2.6/bin).

I don't understand why calling pypy (in $LOCALBASE/bin) via absolute path or $PATH causes it to not find the shared library but calling it from the directly with a relative path works.

Some questions:

  • Is this a bug in pypy or FreeBSD?
  • Do we need to implement some kind of workaround (maybe via some linking voodoo)?
  • Or do we need to use some kind of trampoline code (I have some shell code that sets LD_LIBRARY_PATH)?
robak added a comment.Jul 28 2015, 8:12 AM

Here are the results of my build tests, the third revision:

http://pd.valinor.palantiri.org/data/latest-per-pkg/pypy/2.6.0_2/

It seems to build fine on 10/9 both amd64 and i386. I'll now move to functional testing, namely pip/virtualenv/flask and report the results.

robak added a comment.Jul 28 2015, 8:37 AM

Functional testing isnt possible yet, since I am having the very same issue described by dbn:

root@04-dev:~ # pkg install /tmp/pypy-2.6.0_2.txz
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	pypy: 2.6.0_2

The process will require 85 MiB more space.

Proceed with this action? [y/N]: y
[1/1] Installing pypy-2.6.0_2...
[1/1] Extracting pypy-2.6.0_2: 100%
root@04-dev:~ # rehash
root@04-dev:~ # pypy
Shared object "libpypy-c.so" not found, required by "pypy"
root@04-dev:~ #

Quick googling finds that:

https://bitbucket.org/pypy/pypy/issues/1971/pypy-25-does-not-install-into-virtualenv

and that:

http://pypy.readthedocs.org/en/latest/embedding.html

Perhaps this will give you some hints?

It seems that the libpypy-c.so is installed in wrong place:

root@04-dev:/usr/local/pypy-2.6/bin # ls -la
total 58608
drwxr-xr-x  2 root  wheel       512 Jul 28 13:49 .
drwxr-xr-x  7 root  wheel       512 Jul 28 13:49 ..
-rwxr-xr-x  1 root  wheel  59944720 Jul 28 04:07 libpypy-c.so
-rwxr-xr-x  1 root  wheel      5152 Jul 28 04:07 pypy
root@04-dev:/usr/local/pypy-2.6/bin # ./pypy
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 27 2015, 23:46:30)
[PyPy 2.6.0 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>>
dbn added a comment.Jul 28 2015, 6:27 PM

I really don't understand this bug:

# cd /
# ./usr/local/bin/pypy
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 26 2015, 18:38:23)
[PyPy 2.6.0 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>
# /usr/local/bin/pypy
Shared object "libpypy-c.so" not found, required by "pypy"

I don't even know who to report this to? Why would one way work and not another?

dbn updated this revision to Diff 7438.EditedJul 28 2015, 7:15 PM

Fixing linking issue

I have implemented a workaround for the linking issue:

I created a trampoline shell script that sets LD_LIBRARY_PATH so that pypy can find libpypy-c.so.

Testing virtualenv

I have tested this with virtualenv:

# virtualenv -p `which pypy` pypy
Running virtualenv with interpreter /usr/local/bin/pypy                                                                            
New pypy executable in pypy/bin/pypy                                                                                               
Installing setuptools, pip, wheel...done.

# source pypy/bin/activate.csh

[pypy] # which python
/tmp/pypy/bin/python 

[pypy] # python
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 26 2015, 18:38:23)                                                     
[PyPy 2.6.0 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10                        
Type "help", "copyright", "credits" or "license" for more information.                                                             
>>>>

Outstanding items

  • I have emails hackers/ports requesting help with the underlying issue.
  • The regression-test target still does not run however I don't consider this blocking (the tests have never run properly)

Hi,

Thank you for your work on that port.

According to http://pypy.readthedocs.org/en/latest/embedding.html the shared library is now enabled by default and disabling it will remove support for linking with PyPy from C.
Your problems with linking can be resolved very easily by adding

USE_LDCONFIG= ${PREFIX}/bin

in the Makefile. That'd allow ld.so(8) to find the library without specifying LD_LIBRARY_PATH.
Altho bin isn't standard place from the perspective of FreeBSD but when you chose to use vendor path it should be left that way because all examples on the Internet specify to look for libpypy-c.so in pypy/bin.

I am running a new build testing Pawel's proposed change - once it builds, I'll test it with virtualenv as well and report the results.

Thanks for your input, Pawel!

Obviously it should be

USE_LDCONFIG= ${PREFIX}/${PYPY_DIR}/bin
robak added a comment.Jul 29 2015, 2:07 PM

Can we get a patch without the 'trampoline' thing accomodating USE_LDCONFIG change?

dbn updated this revision to Diff 7480.Jul 29 2015, 7:36 PM

Update with USE_LDCONFIG as suggested by Pawel, thank you!

robak added a comment.Jul 29 2015, 8:18 PM

Latest patch fails to apply cleanly around PORTREVISION and when adjusted manually, the port build fails on patch phase:

=======================<phase: extract        >============================
===>   pypy-2.6.0_2 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by pypy-2.6.0_2 for building
===>  Extracting for pypy-2.6.0_2
=> SHA256 Checksum OK for pypy/release-2.6.0.tar.bz2.
===========================================================================
=======================<phase: patch-depends  >============================
===========================================================================
=======================<phase: patch          >============================
===>  Patching for pypy-2.6.0_2
===>  Applying FreeBSD patches for pypy-2.6.0_2
Ignoring previously applied (or reversed) patch.
1 out of 1 hunks ignored--saving rejects to pypy/tool/release/package.py.rej
=> Patch patch-pypy_tool_release_package.py failed to apply cleanly.
*** [do-patch] Error code 1
dbn added a comment.Jul 29 2015, 8:46 PM

I have double checked the patch - it works for me. Please make sure you have a clean checkout before applying the patch.

I've tested building the latest patch against 9/10 amd64/i38 and the port builds just fine. I've did some functional tests on 10.1 amd64 and here are the results:

Non virtualenv execution:

root@04-dev:~ # pypy
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 30 2015, 02:31:20)
[PyPy 2.6.0] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>>

Virtualenv execution:

r@04-dev:/tmp % virtualenv -p pypy test
Running virtualenv with interpreter /usr/local/bin/pypy
New pypy executable in test/bin/pypy
Installing setuptools, pip, wheel...done.
r@04-dev:/tmp %

Virtualenv activated ldd:

[test] r@04-dev:/tmp % ldd /tmp/test/bin/pypy
/tmp/test/bin/pypy:
       libpypy-c.so => /tmp/test/bin//libpypy-c.so (0x80081d000)
       libthr.so.3 => /lib/libthr.so.3 (0x804337000)
       libc.so.7 => /lib/libc.so.7 (0x80455c000)
       libbz2.so.4 => /usr/lib/libbz2.so.4 (0x804906000)
       libm.so.5 => /lib/libm.so.5 (0x804b18000)
       libintl.so.8 => /usr/local/lib/libintl.so.8 (0x804d40000)
       libexpat.so.1 => /usr/local/lib/libexpat.so.1 (0x804f4a000)
       libz.so.6 => /lib/libz.so.6 (0x805170000)
       libssl.so.7 => /usr/lib/libssl.so.7 (0x805386000)
       libcrypto.so.7 => /lib/libcrypto.so.7 (0x8055f1000)
       libffi.so.6 => /usr/local/lib/libffi.so.6 (0x8059e5000)
       libcrypt.so.5 => /lib/libcrypt.so.5 (0x805bec000)
       librt.so.1 => /usr/lib/librt.so.1 (0x805e0c000)
       libutil.so.9 => /lib/libutil.so.9 (0x806012000)
       libncurses.so.8 => /lib/libncurses.so.8 (0x806224000)
[test] r@04-dev:/tmp %

Non virtualenv ldd:

r@04-dev:/tmp % ldd /usr/local/bin/pypy
/usr/local/bin/pypy:
       libpypy-c.so => /usr/local/pypy-2.6/bin/libpypy-c.so (0x80081d000)
       libthr.so.3 => /lib/libthr.so.3 (0x804337000)
       libc.so.7 => /lib/libc.so.7 (0x80455c000)
       libbz2.so.4 => /usr/lib/libbz2.so.4 (0x804906000)
       libm.so.5 => /lib/libm.so.5 (0x804b18000)
       libintl.so.8 => /usr/local/lib/libintl.so.8 (0x804d40000)
       libexpat.so.1 => /usr/local/lib/libexpat.so.1 (0x804f4a000)
       libz.so.6 => /lib/libz.so.6 (0x805170000)
       libssl.so.7 => /usr/lib/libssl.so.7 (0x805386000)
       libcrypto.so.7 => /lib/libcrypto.so.7 (0x8055f1000)
       libffi.so.6 => /usr/local/lib/libffi.so.6 (0x8059e5000)
       libcrypt.so.5 => /lib/libcrypt.so.5 (0x805bec000)
       librt.so.1 => /usr/lib/librt.so.1 (0x805e0c000)
       libutil.so.9 => /lib/libutil.so.9 (0x806012000)
       libncurses.so.8 => /lib/libncurses.so.8 (0x806224000)
r@04-dev:/tmp %

I've also installed flask with bunch other libraries in virtualenv and tested some web applications this way and so far everything seems to be working as expected.

One thing to note is the double // in the virtualenv activated ldd, but I am not sure if/how bad it is. From my perspective, the ports works far better than before, so I am accepting this revision.

robak accepted this revision.Jul 30 2015, 10:36 AM
robak edited edge metadata.
This revision is now accepted and ready to land.Jul 30 2015, 10:36 AM
This revision was automatically updated to reflect the committed changes.
dbn removed reviewers: alfred, koobs.Aug 1 2015, 9:11 AM

Committed in r393390. Thank you for reviewing :-)