Page MenuHomeFreeBSD

lang/pypy: simplify port
ClosedPublic

Authored by dbn on Jul 26 2015, 8:21 PM.
Tags
Referenced Files
Unknown Object (File)
Wed, Mar 20, 6:13 AM
Unknown Object (File)
Fri, Mar 15, 3:17 AM
Unknown Object (File)
Feb 14 2024, 8:02 PM
Unknown Object (File)
Jan 30 2024, 1:56 PM
Unknown Object (File)
Jan 10 2024, 1:50 AM
Unknown Object (File)
Jan 7 2024, 3:51 PM
Unknown Object (File)
Dec 9 2023, 12:49 AM
Unknown Object (File)
Dec 8 2023, 4:12 PM

Details

Summary
  • Use upstreams directory layout instead of FreeBSD's preferred directory layout [1]
  • Move the pypy installation into $PREFIX/pypy-x.y
  • Remove the ability to build multiple instances (i.e. default to just the JIT instance)
  • Install the binary as pypy (instead of pypy-2.6 with a symlink to pypy)
  • Remove the creation of cffi modules (will be outsourced to other ports, a la cpython)
  • Remove sqlite3 and gdbm dependencies (i.e. cffi modules)

PR: 183795

Test Plan
  • Run the patch through poudriere for a few permutations.
  • Run the regression-test target (currently blocking on hard-coded reference to g++)
  • Bootstap the port [DONE]
  • portlint [DONE]

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

dbn retitled this revision from to lang/pypy: simplify port.
dbn updated this object.
dbn edited the test plan for this revision. (Show Details)
dbn added reviewers: koobs, alfred, robak.
dbn set the repository for this revision to rP FreeBSD ports repository.
dbn added a project: Python.
dbn added a subscriber: Python.

Fix OPTIONS as identified by portlint.

I'll run this through my full 8/9/10/11 i386/amd64 poudriere test suite and will let you know my results. Thanks for fixing that!

The patch fails to apply cleanly:

Patching file files/patch-rpython__rtyper__tool__rffi_platform.py using Plan A...
Hunk #1 succeeded at 0.
Removing files/patch-rpython__rtyper__tool__rffi_platform.py (empty after patching).
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: files/patch-rpython_config_support.py
|===================================================================
|--- files/patch-rpython_config_support.py
|+++ files/patch-rpython_config_support.py
--------------------------
File to patch: files/patch-rpython_config_support.py
No file found--skip this patch? [n]
patch: **** can't find files/patch-rpython_config_support.py
dbn updated this object.
dbn edited the test plan for this revision. (Show Details)
dbn edited edge metadata.
  • Update patch using svn diff --patch-compatible.
  • Fix OPTIONS to actually show option under correct situations
  • Remove '`' around ${WRKDIR}/build.

While failing to build on 8, it does build fine on 10.1 amd64. I am testing the build for 10.1 i386 now, and if that will work, I'll move to testing on 9, and then I'll do tests with virtualenv.

In D3209#64602, @robak wrote:

While failing to build on 8, it does build fine on 10.1 amd64. I am testing the build for 10.1 i386 now, and if that will work, I'll move to testing on 9, and then I'll do tests with virtualenv.

I think you can just ignore 8, its support cycle is less than a week left.

@lwhsu: while I agree FreeBSD 8 can be ignored, this is a bug in the code.

I failed to pass in $MAKE_ENV in the translation step - thus the clang base compiler was being used instead of of $CC. This will work where clang is available (and sufficient) but makes USES=compiler:c11 redundant and fails to honor user requests.

This patch fixes this but should not require a full retest.

I have encountered a strange problem in my testing:

# make install
<snip/>
# pypy
Shared object "libpypy-c.so" not found, required by "pypy"
# cd /usr/local/bin
# ./pypy
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 26 2015, 18:38:23)
[PyPy 2.6.0 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.

The linking error happens irrespective of work directory (i.e. $LOCALBASE/bin or $LOCALBASE/pypy-2.6/bin).

I don't understand why calling pypy (in $LOCALBASE/bin) via absolute path or $PATH causes it to not find the shared library but calling it from the directly with a relative path works.

Some questions:

  • Is this a bug in pypy or FreeBSD?
  • Do we need to implement some kind of workaround (maybe via some linking voodoo)?
  • Or do we need to use some kind of trampoline code (I have some shell code that sets LD_LIBRARY_PATH)?

Here are the results of my build tests, the third revision:

http://pd.valinor.palantiri.org/data/latest-per-pkg/pypy/2.6.0_2/

It seems to build fine on 10/9 both amd64 and i386. I'll now move to functional testing, namely pip/virtualenv/flask and report the results.

Functional testing isnt possible yet, since I am having the very same issue described by dbn:

root@04-dev:~ # pkg install /tmp/pypy-2.6.0_2.txz
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	pypy: 2.6.0_2

The process will require 85 MiB more space.

Proceed with this action? [y/N]: y
[1/1] Installing pypy-2.6.0_2...
[1/1] Extracting pypy-2.6.0_2: 100%
root@04-dev:~ # rehash
root@04-dev:~ # pypy
Shared object "libpypy-c.so" not found, required by "pypy"
root@04-dev:~ #

Quick googling finds that:

https://bitbucket.org/pypy/pypy/issues/1971/pypy-25-does-not-install-into-virtualenv

and that:

http://pypy.readthedocs.org/en/latest/embedding.html

Perhaps this will give you some hints?

It seems that the libpypy-c.so is installed in wrong place:

root@04-dev:/usr/local/pypy-2.6/bin # ls -la
total 58608
drwxr-xr-x  2 root  wheel       512 Jul 28 13:49 .
drwxr-xr-x  7 root  wheel       512 Jul 28 13:49 ..
-rwxr-xr-x  1 root  wheel  59944720 Jul 28 04:07 libpypy-c.so
-rwxr-xr-x  1 root  wheel      5152 Jul 28 04:07 pypy
root@04-dev:/usr/local/pypy-2.6/bin # ./pypy
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 27 2015, 23:46:30)
[PyPy 2.6.0 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>>

I really don't understand this bug:

# cd /
# ./usr/local/bin/pypy
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 26 2015, 18:38:23)
[PyPy 2.6.0 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>
# /usr/local/bin/pypy
Shared object "libpypy-c.so" not found, required by "pypy"

I don't even know who to report this to? Why would one way work and not another?

Fixing linking issue

I have implemented a workaround for the linking issue:

I created a trampoline shell script that sets LD_LIBRARY_PATH so that pypy can find libpypy-c.so.

Testing virtualenv

I have tested this with virtualenv:

# virtualenv -p `which pypy` pypy
Running virtualenv with interpreter /usr/local/bin/pypy                                                                            
New pypy executable in pypy/bin/pypy                                                                                               
Installing setuptools, pip, wheel...done.

# source pypy/bin/activate.csh

[pypy] # which python
/tmp/pypy/bin/python 

[pypy] # python
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 26 2015, 18:38:23)                                                     
[PyPy 2.6.0 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10                        
Type "help", "copyright", "credits" or "license" for more information.                                                             
>>>>

Outstanding items

  • I have emails hackers/ports requesting help with the underlying issue.
  • The regression-test target still does not run however I don't consider this blocking (the tests have never run properly)

Hi,

Thank you for your work on that port.

According to http://pypy.readthedocs.org/en/latest/embedding.html the shared library is now enabled by default and disabling it will remove support for linking with PyPy from C.
Your problems with linking can be resolved very easily by adding

USE_LDCONFIG= ${PREFIX}/bin

in the Makefile. That'd allow ld.so(8) to find the library without specifying LD_LIBRARY_PATH.
Altho bin isn't standard place from the perspective of FreeBSD but when you chose to use vendor path it should be left that way because all examples on the Internet specify to look for libpypy-c.so in pypy/bin.

I am running a new build testing Pawel's proposed change - once it builds, I'll test it with virtualenv as well and report the results.

Thanks for your input, Pawel!

Obviously it should be

USE_LDCONFIG= ${PREFIX}/${PYPY_DIR}/bin

Can we get a patch without the 'trampoline' thing accomodating USE_LDCONFIG change?

Update with USE_LDCONFIG as suggested by Pawel, thank you!

Latest patch fails to apply cleanly around PORTREVISION and when adjusted manually, the port build fails on patch phase:

=======================<phase: extract        >============================
===>   pypy-2.6.0_2 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by pypy-2.6.0_2 for building
===>  Extracting for pypy-2.6.0_2
=> SHA256 Checksum OK for pypy/release-2.6.0.tar.bz2.
===========================================================================
=======================<phase: patch-depends  >============================
===========================================================================
=======================<phase: patch          >============================
===>  Patching for pypy-2.6.0_2
===>  Applying FreeBSD patches for pypy-2.6.0_2
Ignoring previously applied (or reversed) patch.
1 out of 1 hunks ignored--saving rejects to pypy/tool/release/package.py.rej
=> Patch patch-pypy_tool_release_package.py failed to apply cleanly.
*** [do-patch] Error code 1

I have double checked the patch - it works for me. Please make sure you have a clean checkout before applying the patch.

I've tested building the latest patch against 9/10 amd64/i38 and the port builds just fine. I've did some functional tests on 10.1 amd64 and here are the results:

Non virtualenv execution:

root@04-dev:~ # pypy
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jul 30 2015, 02:31:20)
[PyPy 2.6.0] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>>

Virtualenv execution:

r@04-dev:/tmp % virtualenv -p pypy test
Running virtualenv with interpreter /usr/local/bin/pypy
New pypy executable in test/bin/pypy
Installing setuptools, pip, wheel...done.
r@04-dev:/tmp %

Virtualenv activated ldd:

[test] r@04-dev:/tmp % ldd /tmp/test/bin/pypy
/tmp/test/bin/pypy:
       libpypy-c.so => /tmp/test/bin//libpypy-c.so (0x80081d000)
       libthr.so.3 => /lib/libthr.so.3 (0x804337000)
       libc.so.7 => /lib/libc.so.7 (0x80455c000)
       libbz2.so.4 => /usr/lib/libbz2.so.4 (0x804906000)
       libm.so.5 => /lib/libm.so.5 (0x804b18000)
       libintl.so.8 => /usr/local/lib/libintl.so.8 (0x804d40000)
       libexpat.so.1 => /usr/local/lib/libexpat.so.1 (0x804f4a000)
       libz.so.6 => /lib/libz.so.6 (0x805170000)
       libssl.so.7 => /usr/lib/libssl.so.7 (0x805386000)
       libcrypto.so.7 => /lib/libcrypto.so.7 (0x8055f1000)
       libffi.so.6 => /usr/local/lib/libffi.so.6 (0x8059e5000)
       libcrypt.so.5 => /lib/libcrypt.so.5 (0x805bec000)
       librt.so.1 => /usr/lib/librt.so.1 (0x805e0c000)
       libutil.so.9 => /lib/libutil.so.9 (0x806012000)
       libncurses.so.8 => /lib/libncurses.so.8 (0x806224000)
[test] r@04-dev:/tmp %

Non virtualenv ldd:

r@04-dev:/tmp % ldd /usr/local/bin/pypy
/usr/local/bin/pypy:
       libpypy-c.so => /usr/local/pypy-2.6/bin/libpypy-c.so (0x80081d000)
       libthr.so.3 => /lib/libthr.so.3 (0x804337000)
       libc.so.7 => /lib/libc.so.7 (0x80455c000)
       libbz2.so.4 => /usr/lib/libbz2.so.4 (0x804906000)
       libm.so.5 => /lib/libm.so.5 (0x804b18000)
       libintl.so.8 => /usr/local/lib/libintl.so.8 (0x804d40000)
       libexpat.so.1 => /usr/local/lib/libexpat.so.1 (0x804f4a000)
       libz.so.6 => /lib/libz.so.6 (0x805170000)
       libssl.so.7 => /usr/lib/libssl.so.7 (0x805386000)
       libcrypto.so.7 => /lib/libcrypto.so.7 (0x8055f1000)
       libffi.so.6 => /usr/local/lib/libffi.so.6 (0x8059e5000)
       libcrypt.so.5 => /lib/libcrypt.so.5 (0x805bec000)
       librt.so.1 => /usr/lib/librt.so.1 (0x805e0c000)
       libutil.so.9 => /lib/libutil.so.9 (0x806012000)
       libncurses.so.8 => /lib/libncurses.so.8 (0x806224000)
r@04-dev:/tmp %

I've also installed flask with bunch other libraries in virtualenv and tested some web applications this way and so far everything seems to be working as expected.

One thing to note is the double // in the virtualenv activated ldd, but I am not sure if/how bad it is. From my perspective, the ports works far better than before, so I am accepting this revision.

robak edited edge metadata.
This revision is now accepted and ready to land.Jul 30 2015, 10:36 AM
This revision was automatically updated to reflect the committed changes.

Committed in r393390. Thank you for reviewing :-)