New USES+=django and Django default version

[matthew]

Summary

Create a new Mk/Uses/django.mk (largely copied from lua.mk) which adds
a RUN_DEPENDS on the users preferred choice of Django version. Choose
from 1.11, 1.10 or 1.8 currently. Also generate a
DJANGO_PKGNAMEPREFIX which appends the django version to
PYTHON_PKGNAMEPREFIX (so the final result is eg. py27-django118- )

Add a DJANGO_DEFAULT setting to bsd.default-versions.mk

Set the ports default to django-1.11, which is the upstream
recommended default, rather than the current django-1.8

Convert all the ports that have a run-time dependency on Django to
USES+=django

While here convert one port to options helpers and (not really
relevant for this) update another as well as converting to the new
USES+=django

PR: 222724, 224812 (Exp-run)

[ultima]

I'm not positive seahub will run properly on the newest version of django, I know that in the past seahub broke from running a newer version than upstream. databases/py-mysqldb seems to also be broken atm preventing me from doing QA. I'll take a look and see if its a simple fix and do some testing.

[sunpoet]

I would suggest to use USES=django:1.11 instead of USES=django:111 to be consistent with bsd.default-versions.mk and upstream versioning,

Since djano ports cannot coexist, I'm not sure if DJANGO_PKGNAMEPREFIX is necessary.

databases/py-carbon/Makefile

6 ↗

(On Diff #33711)

You need to bump PORTREVISION for the dependency change (from django18 to django111). Other ports may have same issue.

[matthew]

In D12592#261434, @sunpoet wrote:

I would suggest to use USES=django:1.11 instead of USES=django:111 to be consistent with bsd.default-versions.mk and upstream versioning,

Yes -- that's a good idea.

Since djano ports cannot coexist, I'm not sure if DJANGO_PKGNAMEPREFIX is necessary.

Think about installing from package repositories and how things will work in the new flavorful world. This would be better solved by
variable dependencies, but until that lands having multiple packages in the repo that differ only by their dependency tree is the only
available solution.

databases/py-carbon/Makefile

6 ↗

(On Diff #33711)

Yes. PORTREVISION bumps all round.

[mat]

Mk/Uses/django.mk

11 ↗

(On Diff #33711)

I do not think adding python to post is needed, all ports should be using USES=python too.

[matthew]

Mk/Uses/django.mk

11 ↗

(On Diff #33711)

Ah -- OK. That's me not understanding what _USES_POST actually does. I thought it was all about controlling the order that Uses are processed in.

[mat]

Mk/Uses/django.mk

11 ↗

(On Diff #33711)

The Uses are processed in the order they are given.

Uses are invoked a first time during bsd.port.pre.mk, and _USES_POST is used to say that this USES has a "post" section that will be invoked later, in bsd.port.post.mk.

[matthew]

Make version numbers include the decimal point like so:
USES+=django:1.11 -- consistent with DEFAULT_VERSIONS. This also
affects DJANGO_PKGNAMEPREFIX, which now looks like py27-django1.11-

Drop use of _USES_POST

[ultima]

www/seahub/Makefile

36–37

Just finished some testing with Django 1.11, Seahub is not compatible with the newest version and requires 1.8. I tried to change to django:1.8 and the build will fail with
Ignore: Port requires Django version 1.8 which cannot be satisfied by the default version 1.11.

I assume this new USE is being created for the flavor feature and that would probably resolve this issue, though I wonder if there would be conflicts with multiple Django versions installed or if this is even a possible or a desired scenario.

[matthew]

www/seahub/Makefile

36–37

Ah, OK. This should certainly have USES= django:1.8 then

You would also need DEFAULT_VERSIONS+= django=1.8

Yes, I was thinking about how this should work in a FLAVOURed ports tree. Given that seahub depends on a number of other Django modules, to support it properly we'd have to build several FLAVOURS of those modules.

The three different possible versions for each Django port conflict with each other, so if one of your applications requires a particular version of Django, then all of the Django apps on that system would have to use that version.

As it stands, this means that there wouldn't be precompiled Seahub packages available from the default ports repo.

[matthew]

www/seahub requires django-1.8

www/py-jsonfield add CONFLICTS_INSTALL, tracking upstream changes

[sunpoet]

In D12592#262055, @matthew wrote:

Make version numbers include the decimal point like so:
USES+=django:1.11 -- consistent with DEFAULT_VERSIONS. This also
affects DJANGO_PKGNAMEPREFIX, which now looks like py27-django1.11-

Please remove the . in DJANGO_PKGNAMEPREFIX.
That means DJANGO_PKGNAMEPREFIX will look like py27-django111-

BTW, DJANGO_PKGNAMEPREFIX is quite a long prefix (14-15 chars).

[matthew]

Remove the '.' from the Django version in DJANGO_PKGNAMEPREFIX, so it
now looks like: py27-django111-

Fix the CONFLICTS_INSTALL settings in www/py-django-happenings +
www/py-django-mptt; www/py-django-registration +
www/py-django-registration-redux

[matthew]

Merge upstream changes

[matthew]

• Revert accidental inclusion of sysutils/rsyslog8
• Revert accidental inclusion of devel/libfastjson
• Whitespace changes only

[matthew]

Track upstream changes

• Merge branch 'master' into arcpatch-D12592
• Merge branch 'master' into arcpatch-D12592
• Merge branch 'master' into arcpatch-D12592

[FreeBSD_ShaneWare.Biz]

Mk/Uses/django.mk

13 ↗

(On Diff #35451)

py-django20 has landed in ports so 2.0 can be added here.

84 ↗

(On Diff #35451)

Removing '.' leads to DJANGO_VER being 111,110,18,20 - meaning 2.0 will compare as less than 1.10. Could DJANGO_VER be made as three digit? (108,110,111,200)

Maybe add DJANGO_REL as three digit similar to PYTHON_REL.

[mat]

So, this is the time to update this and make it use flavors no ?

[matthew]

In D12592#281340, @mat wrote:

So, this is the time to update this and make it use flavors no ?

Yes, it is. Although I'm a bit short on time right now, I shall be updating this fairly soon.

[matthew]

Merge upstream changes -- ie. package flavours. www/seahub is
problematic, causing the whole poudriere bulk to fail if included in
the list of package targets. Other than that, builds correctly with
both py27 and py36.

Creating django111 and django20 flavours would be interesting, but
that implies packages with multiple flavors which is a problem.

• Use the Django pkgname prefix for dependencies
• Add django-2.0 to the list of possible versions.

ToDo:

fix www/seahub
DJANGO_REL -- release value equivalent to PYTHON_REL

[ultima]

In D12592#283142, @matthew wrote:

Merge upstream changes -- ie. package flavours. www/seahub is
problematic, causing the whole poudriere bulk to fail if included in
the list of package targets. Other than that, builds correctly with
both py27 and py36.

Creating django111 and django20 flavours would be interesting, but
that implies packages with multiple flavors which is a problem.

• Use the Django pkgname prefix for dependencies
• Add django-2.0 to the list of possible versions.

ToDo:

fix www/seahub
DJANGO_REL -- release value equivalent to PYTHON_REL

It maybe time to rename the port to py-seahub. Is this the issue you are running into? Seahub only runs on py27 and django 18. Let me know if there is anything I can assist with.

[matthew]

In D12592#283656, @ultima wrote:

In D12592#283142, @matthew wrote:

Merge upstream changes -- ie. package flavours. www/seahub is
problematic, causing the whole poudriere bulk to fail if included in
the list of package targets. Other than that, builds correctly with
both py27 and py36.

Creating django111 and django20 flavours would be interesting, but
that implies packages with multiple flavors which is a problem.

• Use the Django pkgname prefix for dependencies
• Add django-2.0 to the list of possible versions.

ToDo:

fix www/seahub
DJANGO_REL -- release value equivalent to PYTHON_REL

It maybe time to rename the port to py-seahub. Is this the issue you are running into? Seahub only runs on py27 and django 18. Let me know if there is anything I can assist with.

Well, given the dependencies on both Django and Python, by rights that should be www/py-django-seahub. That however is not precisely the issue. www/seahub builds fine using 'poudriere testbuild' but not when using 'poudriere bulk' -- this is with python=2.7 and django=1.8 in DEFAULT_VERSIONS. It's not that the www/seahub port fails under 'poudriere bulk'. The entire poudriere run aborts without building anything like so:

lucid-nonsense:...work/github/freebsd-ports:# poudriere bulk -j 11_1a -z development -p github -f ~matthew/work/django-port-list
[00:00:00] Creating the reference jail... done
[00:00:00] Mounting system devices for 11_1a-github-development
[00:00:00] Mounting ports/packages/distfiles
[00:00:00] Stashing existing package repository
[00:00:00] Mounting ccache from: /var/cache/ccache
[00:00:00] Mounting packages from: /usr/local/poudriere/data/packages/11_1a-github-development
[00:00:00] Copying /var/db/ports from: /usr/local/etc/poudriere.d/development-options
[00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/development-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/11_1a-github-development/ref/etc/resolv.conf
[00:00:00] Starting jail 11_1a-github-development
[00:00:00] Logs: /usr/local/poudriere/data/logs/bulk/11_1a-github-development/2017-12-21_10h19m34s
[00:00:00] Loading MOVED
[00:00:00] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:00] Gathering ports metadata
[00:00:11] Calculating ports order and dependencies
[00:00:11] Error: compute_deps_pkg failed to lookup pkgname for www/py-django@ processing package seahub-6.2.4_1 from www/seahub -- Does www/py-django provide the '' FLAVOR?
[00:00:12] Error: Fatal errors encountered calculating dependencies
[00:00:12] Cleaning up
11_1a-github-development: removed
11_1a-github-development-n: removed
[00:00:12] Unmounting file systems

[mat]

In D12592#283852, @matthew wrote:

[00:00:11] Error: compute_deps_pkg failed to lookup pkgname for www/py-django@ processing package seahub-6.2.4_1 from www/seahub -- Does www/py-django provide the '' FLAVOR?

This means that www/seahub does not have any flavors, so the RUN_DEPENDS in django.mk ends with ...py-django@ which is interpreted as an empty string flavor. You probably need to replace this with PY_FLAVOR.

[matthew]

In D12592#283856, @mat wrote:

This means that www/seahub does not have any flavors, so the RUN_DEPENDS in django.mk ends with ...py-django@ which is interpreted as an empty string flavor. You probably need to replace this with PY_FLAVOR.

Ahah! Yes, that solves the problem.

[matthew]

Major rework of the code, modelled on python.mk

• FLAVOR support
• Now requires python.mk to preceed djano.mk in the USES definition
• Add a script 'version-rel.sh' to turn typical multipart version numbers into sortable integers
• Add Makefile.version to each of the Django ports

This mostly works, but is not yet quite ready for primetime -- test
building all of the Django related ports results with all the
different flavors generates about 700 out of 800 ports correctly.

[matthew]

Fix remaining build problems. Every Django related port now builds
correctly.

Fixes here involve redefining a number of python related variables
using the Python versions determined from the flavoring. This seems
inelegant -- in a couple of places it's literally cut'n'paste from
python.mk -- but it works!

[matthew]

• Merge branch 'master' into arcpatch-D12592

[matthew]

• Merge branch 'master' into arcpatch-D12592

[mat]

Mk/Uses/django.mk

202 ↗

(On Diff #37398)

I do not see this FLAVORS variable defined beforehand, where does it come from exactly ?

[matthew]

Mk/Uses/django.mk

202 ↗

(On Diff #37398)

That's what is set in python.mk

[matthew]

• Merge branch 'master' into arcpatch-D12592

[mat]

Mk/Uses/django.mk

202 ↗

(On Diff #37398)

Mmm, but django < python, you can absolutely assume that the sort police will put django before python and it will break. You need to add a .include "${USESDIR}/python.mk" before you use any variables coming from python.mk.

[matthew]

Mk/Uses/django.mk

202 ↗

(On Diff #37398)

Agreed that there will be some people who sort the USES line, but not for long. That's why I put in the check for .if !defined(_INCLUDE_USES_PYTHON_MK) at line 70 -- all they'll get is an error message which, I trust, will help to educate them that sorting things is not always a good idea.

Notice that the body of this file is in the .else branch of that test, so none of the python variables will be processed unless python.mk has already been included.

If I include ${USESDIR}/python.mk doesn't that break processing any arguments for the python USES?

[matthew]

• Merge branch 'master' into arcpatch-D12592
• Add new py-django-netfields port

[mat]

Mk/Uses/django.mk

202 ↗

(On Diff #37398)

USES must work the same way whatever their order is.

USES are self contained, and many include other because of dependencies. It should not break anything.

[matthew]

Mk/Uses/django.mk

202 ↗

(On Diff #37398)

That's a problem when you've got several different USES each trying to define FLAVOR and FLAVORS -- it's presumably the last-added USES that wins, which breaks the requirement about invariance to ordering.

I'm thinking we need a rule that any variables that a USES .mk file defines should have a distinct namespace eg. PY_FLAVOR, PY_FLAVORS; DJANGO_FLAVOR, DJANGO_FLAVORS.

For flavour handling specifically, the USES could append their namespace prefix to a generic variable, say FLAVOR_NAMESPACES+=PY and then a separate block of code in bsd.ports.mk to combine the flavors from the different namespaces (sorted alphabetically), with a mechanism to filter out incompatible combinations (eg. py27 doesn't mix with django20). Individual ports could override this mechanism by setting the un-namespaced variables FLAVOR or FLAVORS as now.

[antoine]

Wouldn't it be easier to have USE_PYTHON+= django:somearg ?

[matthew]

Yeah. This approach is a non-starter. Adding django bits to python.mk is probably a better route as Antoine suggests,
but that'sbasically a start-over.

[matthew]

• Merge branch 'master' into arcpatch-D12592
• Remove leftovers from incomplete merging
• Merge branch 'master' into arcpatch-D12592
• Remove the py-dj21-django-foo port variants, as these will now be
• Merge branch 'master' into arcpatch-D12592
• databases/py-carbon no loger has any django dependencies.
• Update Makefile.version in each of the main django ports.
• Survey the tree to identify all of the Django related ports.
• USE_PYTHON=django:1.11+ is identical to the default USES_PYTHON=django
• Merge branch 'master' into arcpatch-D12592
• Merge branch 'master' into arcpatch-D12592
• Django-flavourize newly added www/py-django-cron port
• Fix a mis-merge. Mk/Scripts/version-rel.sh was removed from the
• DJango-2.1 now in the tree
• Reduce the diff to the master branch
• Merge branch 'master' into arcpatch-D12592
• py-dj21-djangoql -> py-djangoql, flavoured for django 2.1+
• Merge branch 'master' into arcpatch-D12592
• Merge branch 'master' into arcpatch-D12592
• The code at line 299 of Mk/Uses/python.mk already parses an extra
• Merge branch 'master' into arcpatch-D12592
• New upstream Django 2.2 -- py-django22 port added.
• Merge branch 'master' into arcpatch-D12592
• Start by defining the outputs -- exported variables and added dependencies.
• Merge branch 'master' into arcpatch-D12592
• Two more Django ports
• Another pass through all the Django ports, switching to the
• Generate the django version as a 5 digit integer
• Merge branch 'master' into arcpatch-D12592
• Merge branch 'master' into arcpatch-D12592
• Django 1.8 was dropped from the ports.
• Merge branch 'master' into arcpatch-D12592
• Merge branch 'master' into arcpatch-D12592
• py-logan needs a TEST_DEPENDS on django
• Delete extraneous blank line
• Generate a 3 digit number from the version for comparison purposes.
• Merge branch 'master' into arcpatch-D12592
• Merge branch 'master' into arcpatch-D12592
• Check that the specified Django version fits within any version
• Merge branch 'master' into arcpatch-D12592
• Regenerate FLAVORS adding Django version information.
• Remove the separate django.mk now that the functionality has been
• Add missing '.'
• Initialise DJANGO_PORTSDIR before referencing it.
• Merge branch 'master' into arcpatch-D12592
• Add django to the list of valid categories
• Revert the approach of using PY_FLAVOR instead of FLAVOR -- FLAVOR is
• Merge branch 'master' into arcpatch-D12592
• Have only one USE_PYTHON line for best results
• Multiple debug fixes
• Adjust CONFLICTS_INSTALL
• PyPi claims that the maximum supported branch of Django for
• Merge branch 'master' into arcpatch-D12592
• Despite the name, django-modelcluster has no dependency on django, so
• Where django is only used as a test dependency, the FLAVOR settings
• Whitespace
• Merge branch 'master' into arcpatch-D12592
• With USE_PYTHON=django, the string 'django' will always appear in
• Merge branch 'master' into arcpatch-D12592
• Set FLAVORS immediately.
• Merge branch 'master' into arcpatch-D12592
• Merge branch 'master' into arcpatch-D12592
• Merge branch 'master' into arcpatch-D12592

This is very much a work in progress and not at all useable yet.

[matthew]

• Django 2.1 is gone, django 3.0 is here.

[matthew]

• Merge py-django-rq and py-dj22-django-rq
• Tidy up cases where django is bought in as a test dependency
• Don't support nodjangoflavor as an option any more.

[mat]

Mk/Uses/python.mk

276

This says SUFFIX, should be PREFIX.

316–318

Can you try to only change what you need, adding whitespace and rewrapping differently is not great.

704

By definition, the default flavor is the first of the flavors.

.  if empty(FLAVOR)
FLAVOR:=        ${FLAVORS:[1]}
.  endif

[matthew]

Mk/Uses/python.mk

276

This simply parallels the comment on PYTHON_PKGNAMEPREFIX at lines 242-245:

# PYTHON_PKGNAMEPREFIX
#			- Use this as a ${PKGNAMEPREFIX} to distinguish
#			  packages for different Python versions.
#			  default: py${PYTHON_SUFFIX}-

${PYTHON_SUFFIX} is maybe a badly named variable, but changing it would be a lot of churn.

[matthew]

• Don't re-wrap _VALID_PYTHON_FEATURES
• The default flavor is the first of the flavours.
• Merge branch 'master' into arcpatch-D12592

[matthew]

Diff against master

[matthew]

• Merge branch 'master' into arcpatch-D12592

[matthew]

• Merge pull request #1 from knobix/arcpatch-D12592

[FreeBSD_ShaneWare.Biz]

While there has been a comment on the length of DJANGO_PKGNAMEPREFIX no one seems to have suggested shortening it.

py37-django30-portname

could be shortened to

py37-dj30-portname

port folder names could be similar - py-dj-portname

[matthew]

In D12592#524776, @FreeBSD_ShaneWare.Biz wrote:

While there has been a comment on the length of DJANGO_PKGNAMEPREFIX no one seems to have suggested shortening it.

py37-django30-portname

could be shortened to

py37-dj30-portname

This is certainly something to consider -- many Django modules already have 'django' in their name, so having it twice in the package name is pretty redundant.

port folder names could be similar - py-dj-portname

However, I don't see much value in this. As I said, most Django-related ports already self-identify as such. This patch already makes hundreds of changes to the tree, and I don't think adding quite a lot of additional churn on top of that, to no real benefit, would be a good idea.

[matthew]

This is well out of date.