Page MenuHomeFreeBSD

New USES+=django and Django default version
AbandonedPublic

Authored by matthew on Oct 4 2017, 10:23 PM.

Details

Reviewers
ultima
Group Reviewers
O5: Ports Framework(Owns No Changed Paths)
portmgr
Python
Summary

Create a new Mk/Uses/django.mk (largely copied from lua.mk) which adds
a RUN_DEPENDS on the users preferred choice of Django version. Choose
from 1.11, 1.10 or 1.8 currently. Also generate a
DJANGO_PKGNAMEPREFIX which appends the django version to
PYTHON_PKGNAMEPREFIX (so the final result is eg. py27-django118- )

Add a DJANGO_DEFAULT setting to bsd.default-versions.mk

Set the ports default to django-1.11, which is the upstream
recommended default, rather than the current django-1.8

Convert all the ports that have a run-time dependency on Django to
USES+=django

While here convert one port to options helpers and (not really
relevant for this) update another as well as converting to the new
USES+=django

PR: 222724, 224812 (Exp-run)

Diff Detail

Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 14059
Build 14243: arc lint + arc unit

Event Timeline

matthew created this revision.Oct 4 2017, 10:23 PM

Note: lua.mk is not a good example to follow, there are some bugs in it...

antoine added inline comments.Oct 4 2017, 10:37 PM
Mk/Uses/django.mk
75

This is a bug, if it can't find a wanted version, it should prefer the default version instead of the highest one

matthew added inline comments.Oct 4 2017, 11:00 PM
Mk/Uses/django.mk
75

Yes, this is wrong, but not in the way you state. This is trying to handle the situation where eg. DEFAULT_VERSIONS contains django=1.8 but the port says something like USES=django:110+

In this case I should probably just set IGNORE

antoine added inline comments.Oct 5 2017, 5:42 AM
Mk/Uses/django.mk
75

If you USES=django:18+ it will choose the highest version too instead of using the default one

matthew updated this revision to Diff 33709.Oct 5 2017, 6:23 AM

Address antoine's comments. Rework django.mk such that the task is
now to identify if the Django default version is one allowed by the
port. If not, then set IGNORE for the port.

matthew marked 2 inline comments as done.Oct 5 2017, 6:26 AM
matthew added inline comments.
Mk/Uses/django.mk
75

Now it will choose ither the default version, or the port will be ignored.

matthew updated this revision to Diff 33710.Oct 5 2017, 6:32 AM
matthew marked an inline comment as done.

Remove some additional changes and a new port outside the scope of
this review.

matthew updated this revision to Diff 33711.Oct 5 2017, 6:38 AM

security/py-first-server needs a more recent version of Django

matthew edited the summary of this revision. (Show Details)Oct 5 2017, 8:49 AM
ultima added a comment.Oct 6 2017, 4:05 AM

I'm not positive seahub will run properly on the newest version of django, I know that in the past seahub broke from running a newer version than upstream. databases/py-mysqldb seems to also be broken atm preventing me from doing QA. I'll take a look and see if its a simple fix and do some testing.

sunpoet added a subscriber: sunpoet.Oct 6 2017, 9:27 AM

I would suggest to use USES=django:1.11 instead of USES=django:111 to be consistent with bsd.default-versions.mk and upstream versioning,

Since djano ports cannot coexist, I'm not sure if DJANGO_PKGNAMEPREFIX is necessary.

databases/py-carbon/Makefile
5–6

You need to bump PORTREVISION for the dependency change (from django18 to django111). Other ports may have same issue.

I would suggest to use USES=django:1.11 instead of USES=django:111 to be consistent with bsd.default-versions.mk and upstream versioning,

Yes -- that's a good idea.

Since djano ports cannot coexist, I'm not sure if DJANGO_PKGNAMEPREFIX is necessary.

Think about installing from package repositories and how things will work in the new flavorful world. This would be better solved by
variable dependencies, but until that lands having multiple packages in the repo that differ only by their dependency tree is the only
available solution.

databases/py-carbon/Makefile
5–6

Yes. PORTREVISION bumps all round.

mat added inline comments.Oct 6 2017, 3:56 PM
Mk/Uses/django.mk
12

I do not think adding python to post is needed, all ports should be using USES=python too.

matthew added inline comments.Oct 6 2017, 4:18 PM
Mk/Uses/django.mk
12

Ah -- OK. That's me not understanding what _USES_POST actually does. I thought it was all about controlling the order that Uses are processed in.

mat added inline comments.Oct 6 2017, 4:30 PM
Mk/Uses/django.mk
12

The Uses are processed in the order they are given.

Uses are invoked a first time during bsd.port.pre.mk, and _USES_POST is used to say that this USES has a "post" section that will be invoked later, in bsd.port.post.mk.

matthew updated this revision to Diff 33821.Oct 8 2017, 5:17 PM

Make version numbers include the decimal point like so:
USES+=django:1.11 -- consistent with DEFAULT_VERSIONS. This also
affects DJANGO_PKGNAMEPREFIX, which now looks like py27-django1.11-

Drop use of _USES_POST

matthew marked 5 inline comments as done.Oct 8 2017, 5:19 PM
ultima added inline comments.Oct 9 2017, 9:31 PM
www/seahub/Makefile
35

Just finished some testing with Django 1.11, Seahub is not compatible with the newest version and requires 1.8. I tried to change to django:1.8 and the build will fail with
Ignore: Port requires Django version 1.8 which cannot be satisfied by the default version 1.11.

I assume this new USE is being created for the flavor feature and that would probably resolve this issue, though I wonder if there would be conflicts with multiple Django versions installed or if this is even a possible or a desired scenario.

matthew added inline comments.Oct 10 2017, 6:13 AM
www/seahub/Makefile
35

Ah, OK. This should certainly have USES= django:1.8 then

You would also need DEFAULT_VERSIONS+= django=1.8

Yes, I was thinking about how this should work in a FLAVOURed ports tree. Given that seahub depends on a number of other Django modules, to support it properly we'd have to build several FLAVOURS of those modules.

The three different possible versions for each Django port conflict with each other, so if one of your applications requires a particular version of Django, then all of the Django apps on that system would have to use that version.

As it stands, this means that there wouldn't be precompiled Seahub packages available from the default ports repo.

matthew updated this revision to Diff 33860.Oct 10 2017, 6:38 AM

www/seahub requires django-1.8

www/py-jsonfield add CONFLICTS_INSTALL, tracking upstream changes

matthew marked 2 inline comments as done.Oct 10 2017, 6:39 AM

Make version numbers include the decimal point like so:
USES+=django:1.11 -- consistent with DEFAULT_VERSIONS. This also
affects DJANGO_PKGNAMEPREFIX, which now looks like py27-django1.11-

Please remove the . in DJANGO_PKGNAMEPREFIX.
That means DJANGO_PKGNAMEPREFIX will look like py27-django111-

BTW, DJANGO_PKGNAMEPREFIX is quite a long prefix (14-15 chars).

matthew updated this revision to Diff 34003.Oct 15 2017, 9:58 AM

Remove the '.' from the Django version in DJANGO_PKGNAMEPREFIX, so it
now looks like: py27-django111-

Fix the CONFLICTS_INSTALL settings in www/py-django-happenings +
www/py-django-mptt; www/py-django-registration +
www/py-django-registration-redux

matthew updated this revision to Diff 34204.Oct 21 2017, 6:49 AM

Merge upstream changes

matthew updated this revision to Diff 34206.Oct 21 2017, 7:20 AM
  • Revert accidental inclusion of sysutils/rsyslog8
  • Revert accidental inclusion of devel/libfastjson
  • Whitespace changes only
matthew updated this revision to Diff 35451.Nov 19 2017, 11:06 AM

Track upstream changes

  • Merge branch 'master' into arcpatch-D12592
  • Merge branch 'master' into arcpatch-D12592
  • Merge branch 'master' into arcpatch-D12592
FreeBSD_ShaneWare.Biz added inline comments.
Mk/Uses/django.mk
14

py-django20 has landed in ports so 2.0 can be added here.

85

Removing '.' leads to DJANGO_VER being 111,110,18,20 - meaning 2.0 will compare as less than 1.10. Could DJANGO_VER be made as three digit? (108,110,111,200)

Maybe add DJANGO_REL as three digit similar to PYTHON_REL.

mat added a comment.Dec 12 2017, 12:33 PM

So, this is the time to update this and make it use flavors no ?

In D12592#281340, @mat wrote:

So, this is the time to update this and make it use flavors no ?

Yes, it is. Although I'm a bit short on time right now, I shall be updating this fairly soon.

matthew updated this revision to Diff 36754.Dec 19 2017, 6:43 AM

Merge upstream changes -- ie. package flavours. www/seahub is
problematic, causing the whole poudriere bulk to fail if included in
the list of package targets. Other than that, builds correctly with
both py27 and py36.

Creating django111 and django20 flavours would be interesting, but
that implies packages with multiple flavors which is a problem.

  • Use the Django pkgname prefix for dependencies
  • Add django-2.0 to the list of possible versions.

ToDo:

fix www/seahub
DJANGO_REL -- release value equivalent to PYTHON_REL

Merge upstream changes -- ie. package flavours. www/seahub is
problematic, causing the whole poudriere bulk to fail if included in
the list of package targets. Other than that, builds correctly with
both py27 and py36.
Creating django111 and django20 flavours would be interesting, but
that implies packages with multiple flavors which is a problem.

  • Use the Django pkgname prefix for dependencies
  • Add django-2.0 to the list of possible versions.

ToDo:

fix www/seahub
DJANGO_REL -- release value equivalent to PYTHON_REL

It maybe time to rename the port to py-seahub. Is this the issue you are running into? Seahub only runs on py27 and django 18. Let me know if there is anything I can assist with.

Merge upstream changes -- ie. package flavours. www/seahub is
problematic, causing the whole poudriere bulk to fail if included in
the list of package targets. Other than that, builds correctly with
both py27 and py36.
Creating django111 and django20 flavours would be interesting, but
that implies packages with multiple flavors which is a problem.

  • Use the Django pkgname prefix for dependencies
  • Add django-2.0 to the list of possible versions.

ToDo:

fix www/seahub
DJANGO_REL -- release value equivalent to PYTHON_REL

It maybe time to rename the port to py-seahub. Is this the issue you are running into? Seahub only runs on py27 and django 18. Let me know if there is anything I can assist with.

Well, given the dependencies on both Django and Python, by rights that should be www/py-django-seahub. That however is not precisely the issue. www/seahub builds fine using 'poudriere testbuild' but not when using 'poudriere bulk' -- this is with python=2.7 and django=1.8 in DEFAULT_VERSIONS. It's not that the www/seahub port fails under 'poudriere bulk'. The entire poudriere run aborts without building anything like so:

lucid-nonsense:...work/github/freebsd-ports:# poudriere bulk -j 11_1a -z development -p github -f ~matthew/work/django-port-list
[00:00:00] Creating the reference jail... done
[00:00:00] Mounting system devices for 11_1a-github-development
[00:00:00] Mounting ports/packages/distfiles
[00:00:00] Stashing existing package repository
[00:00:00] Mounting ccache from: /var/cache/ccache
[00:00:00] Mounting packages from: /usr/local/poudriere/data/packages/11_1a-github-development
[00:00:00] Copying /var/db/ports from: /usr/local/etc/poudriere.d/development-options
[00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/development-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/11_1a-github-development/ref/etc/resolv.conf
[00:00:00] Starting jail 11_1a-github-development
[00:00:00] Logs: /usr/local/poudriere/data/logs/bulk/11_1a-github-development/2017-12-21_10h19m34s
[00:00:00] Loading MOVED
[00:00:00] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:00] Gathering ports metadata
[00:00:11] Calculating ports order and dependencies
[00:00:11] Error: compute_deps_pkg failed to lookup pkgname for www/py-django@ processing package seahub-6.2.4_1 from www/seahub -- Does www/py-django provide the '' FLAVOR?
[00:00:12] Error: Fatal errors encountered calculating dependencies
[00:00:12] Cleaning up
11_1a-github-development: removed
11_1a-github-development-n: removed
[00:00:12] Unmounting file systems
mat added a comment.Dec 21 2017, 1:32 PM

[00:00:11] Error: compute_deps_pkg failed to lookup pkgname for www/py-django@ processing package seahub-6.2.4_1 from www/seahub -- Does www/py-django provide the '' FLAVOR?

This means that www/seahub does not have any flavors, so the RUN_DEPENDS in django.mk ends with ...py-django@ which is interpreted as an empty string flavor. You probably need to replace this with PY_FLAVOR.

In D12592#283856, @mat wrote:

This means that www/seahub does not have any flavors, so the RUN_DEPENDS in django.mk ends with ...py-django@ which is interpreted as an empty string flavor. You probably need to replace this with PY_FLAVOR.

Ahah! Yes, that solves the problem.

matthew updated this revision to Diff 37299.Dec 31 2017, 9:46 AM

Major rework of the code, modelled on python.mk

  • FLAVOR support
  • Now requires python.mk to preceed djano.mk in the USES definition
  • Add a script 'version-rel.sh' to turn typical multipart version numbers into sortable integers
  • Add Makefile.version to each of the Django ports

This mostly works, but is not yet quite ready for primetime -- test
building all of the Django related ports results with all the
different flavors generates about 700 out of 800 ports correctly.

matthew updated this revision to Diff 37330.Dec 31 2017, 9:21 PM

Fix remaining build problems. Every Django related port now builds
correctly.

Fixes here involve redefining a number of python related variables
using the Python versions determined from the flavoring. This seems
inelegant -- in a couple of places it's literally cut'n'paste from
python.mk -- but it works!

matthew marked 3 inline comments as done.Dec 31 2017, 9:26 PM
matthew edited the summary of this revision. (Show Details)Jan 1 2018, 8:26 AM
matthew marked an inline comment as done.Jan 1 2018, 8:28 AM
matthew updated this revision to Diff 37373.Jan 1 2018, 9:00 PM
  • Merge branch 'master' into arcpatch-D12592
matthew updated this revision to Diff 37398.Jan 2 2018, 6:51 AM
  • Merge branch 'master' into arcpatch-D12592
mat added inline comments.Jan 2 2018, 2:38 PM
Mk/Uses/django.mk
202

I do not see this FLAVORS variable defined beforehand, where does it come from exactly ?

matthew added inline comments.Jan 2 2018, 4:14 PM
Mk/Uses/django.mk
202

That's what is set in python.mk

matthew updated this revision to Diff 37445.Jan 3 2018, 6:35 AM
  • Merge branch 'master' into arcpatch-D12592
mat added inline comments.Jan 3 2018, 1:08 PM
Mk/Uses/django.mk
202

Mmm, but django < python, you can absolutely assume that the sort police will put django before python and it will break. You need to add a .include "${USESDIR}/python.mk" before you use any variables coming from python.mk.

matthew added inline comments.Jan 3 2018, 6:59 PM
Mk/Uses/django.mk
202

Agreed that there will be some people who sort the USES line, but not for long. That's why I put in the check for .if !defined(_INCLUDE_USES_PYTHON_MK) at line 70 -- all they'll get is an error message which, I trust, will help to educate them that sorting things is not always a good idea.

Notice that the body of this file is in the .else branch of that test, so none of the python variables will be processed unless python.mk has already been included.

If I include ${USESDIR}/python.mk doesn't that break processing any arguments for the python USES?

matthew updated this revision to Diff 37499.Jan 4 2018, 7:30 AM
  • Merge branch 'master' into arcpatch-D12592
  • Add new py-django-netfields port
mat added inline comments.Jan 4 2018, 12:27 PM
Mk/Uses/django.mk
202

USES must work the same way whatever their order is.

USES are self contained, and many include other because of dependencies. It should not break anything.

matthew added inline comments.Jan 5 2018, 7:13 AM
Mk/Uses/django.mk
202

That's a problem when you've got several different USES each trying to define FLAVOR and FLAVORS -- it's presumably the last-added USES that wins, which breaks the requirement about invariance to ordering.

I'm thinking we need a rule that any variables that a USES .mk file defines should have a distinct namespace eg. PY_FLAVOR, PY_FLAVORS; DJANGO_FLAVOR, DJANGO_FLAVORS.

For flavour handling specifically, the USES could append their namespace prefix to a generic variable, say FLAVOR_NAMESPACES+=PY and then a separate block of code in bsd.ports.mk to combine the flavors from the different namespaces (sorted alphabetically), with a mechanism to filter out incompatible combinations (eg. py27 doesn't mix with django20). Individual ports could override this mechanism by setting the un-namespaced variables FLAVOR or FLAVORS as now.

Wouldn't it be easier to have USE_PYTHON+= django:somearg ?

matthew abandoned this revision.Mar 31 2018, 5:49 PM

Yeah. This approach is a non-starter. Adding django bits to python.mk is probably a better route as Antoine suggests,
but that'sbasically a start-over.

leres added a subscriber: leres.Mar 9 2019, 5:59 PM