Details

Reviewers

ultima
kai
sbz

Group Reviewers

O5: Ports Framework	(Owns No Changed Paths)
portmgr
Python

Summary

Create a new Mk/Uses/django.mk (largely copied from lua.mk) which adds
a RUN_DEPENDS on the users preferred choice of Django version. Choose
from 1.11, 1.10 or 1.8 currently. Also generate a
DJANGO_PKGNAMEPREFIX which appends the django version to
PYTHON_PKGNAMEPREFIX (so the final result is eg. py27-django118- )

Add a DJANGO_DEFAULT setting to bsd.default-versions.mk

Set the ports default to django-1.11, which is the upstream
recommended default, rather than the current django-1.8

Convert all the ports that have a run-time dependency on Django to
USES+=django

While here convert one port to options helpers and (not really
relevant for this) update another as well as converting to the new
USES+=django

PR: 222724, 224812 (Exp-run)

Diff Detail

Repository

rP FreeBSD ports repository

Lint

No Lint Coverage

Unit

No Test Coverage

Build Status

Buildable 11983
Build 12307: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

matthew added a reviewer: ultima.Oct 5 2017, 6:45 AM

matthew edited the summary of this revision. (Show Details)Oct 5 2017, 8:49 AM

I'm not positive seahub will run properly on the newest version of django, I know that in the past seahub broke from running a newer version than upstream. databases/py-mysqldb seems to also be broken atm preventing me from doing QA. I'll take a look and see if its a simple fix and do some testing.

I would suggest to use USES=django:1.11 instead of USES=django:111 to be consistent with bsd.default-versions.mk and upstream versioning,

Since djano ports cannot coexist, I'm not sure if DJANGO_PKGNAMEPREFIX is necessary.

databases/py-carbon/Makefile
6	You need to bump PORTREVISION for the dependency change (from django18 to django111). Other ports may have same issue.

In D12592#261434, @sunpoet wrote:

I would suggest to use USES=django:1.11 instead of USES=django:111 to be consistent with bsd.default-versions.mk and upstream versioning,

Yes -- that's a good idea.

Since djano ports cannot coexist, I'm not sure if DJANGO_PKGNAMEPREFIX is necessary.

Think about installing from package repositories and how things will work in the new flavorful world. This would be better solved by
variable dependencies, but until that lands having multiple packages in the repo that differ only by their dependency tree is the only
available solution.

databases/py-carbon/Makefile
6	Yes. PORTREVISION bumps all round.

mat added inline comments.Oct 6 2017, 3:56 PM

Mk/Uses/django.mk
12	I do not think adding python to post is needed, all ports should be using USES=python too.

matthew added inline comments.Oct 6 2017, 4:18 PM

Mk/Uses/django.mk
12	Ah -- OK. That's me not understanding what _USES_POST actually does. I thought it was all about controlling the order that Uses are processed in.

mat added inline comments.Oct 6 2017, 4:30 PM

Mk/Uses/django.mk
12	The Uses are processed in the order they are given. Uses are invoked a first time during bsd.port.pre.mk, and _USES_POST is used to say that this USES has a "post" section that will be invoked later, in bsd.port.post.mk.

Make version numbers include the decimal point like so:
USES+=django:1.11 -- consistent with DEFAULT_VERSIONS. This also
affects DJANGO_PKGNAMEPREFIX, which now looks like py27-django1.11-

Drop use of _USES_POST

matthew marked 5 inline comments as done.Oct 8 2017, 5:19 PM

ultima added inline comments.Oct 9 2017, 9:31 PM

www/seahub/Makefile
34	Just finished some testing with Django 1.11, Seahub is not compatible with the newest version and requires 1.8. I tried to change to django:1.8 and the build will fail with Ignore: Port requires Django version 1.8 which cannot be satisfied by the default version 1.11. I assume this new USE is being created for the flavor feature and that would probably resolve this issue, though I wonder if there would be conflicts with multiple Django versions installed or if this is even a possible or a desired scenario.

matthew added inline comments.Oct 10 2017, 6:13 AM

www/seahub/Makefile
34	Ah, OK. This should certainly have USES= django:1.8 then You would also need DEFAULT_VERSIONS+= django=1.8 Yes, I was thinking about how this should work in a FLAVOURed ports tree. Given that seahub depends on a number of other Django modules, to support it properly we'd have to build several FLAVOURS of those modules. The three different possible versions for each Django port conflict with each other, so if one of your applications requires a particular version of Django, then all of the Django apps on that system would have to use that version. As it stands, this means that there wouldn't be precompiled Seahub packages available from the default ports repo.

www/seahub requires django-1.8

www/py-jsonfield add CONFLICTS_INSTALL, tracking upstream changes

matthew marked 2 inline comments as done.Oct 10 2017, 6:39 AM

In D12592#262055, @matthew wrote:

Make version numbers include the decimal point like so:
USES+=django:1.11 -- consistent with DEFAULT_VERSIONS. This also
affects DJANGO_PKGNAMEPREFIX, which now looks like py27-django1.11-

Please remove the . in DJANGO_PKGNAMEPREFIX.
That means DJANGO_PKGNAMEPREFIX will look like py27-django111-

BTW, DJANGO_PKGNAMEPREFIX is quite a long prefix (14-15 chars).

Remove the '.' from the Django version in DJANGO_PKGNAMEPREFIX, so it
now looks like: py27-django111-

Fix the CONFLICTS_INSTALL settings in www/py-django-happenings +
www/py-django-mptt; www/py-django-registration +
www/py-django-registration-redux

Merge upstream changes

Revert accidental inclusion of sysutils/rsyslog8
Revert accidental inclusion of devel/libfastjson
Whitespace changes only

Track upstream changes

Merge branch 'master' into arcpatch-D12592
Merge branch 'master' into arcpatch-D12592
Merge branch 'master' into arcpatch-D12592

Harbormaster completed remote builds in B12856: Diff 35451.Nov 19 2017, 11:06 AM

FreeBSD_ShaneWare.Biz added a subscriber: FreeBSD_ShaneWare.Biz.Dec 12 2017, 12:22 PM

FreeBSD_ShaneWare.Biz added inline comments.

Mk/Uses/django.mk
14	py-django20 has landed in ports so 2.0 can be added here.
85	Removing '.' leads to DJANGO_VER being 111,110,18,20 - meaning 2.0 will compare as less than 1.10. Could DJANGO_VER be made as three digit? (108,110,111,200) Maybe add DJANGO_REL as three digit similar to PYTHON_REL.

So, this is the time to update this and make it use flavors no ?

In D12592#281340, @mat wrote:

So, this is the time to update this and make it use flavors no ?

Yes, it is. Although I'm a bit short on time right now, I shall be updating this fairly soon.

Merge upstream changes -- ie. package flavours. www/seahub is
problematic, causing the whole poudriere bulk to fail if included in
the list of package targets. Other than that, builds correctly with
both py27 and py36.

Creating django111 and django20 flavours would be interesting, but
that implies packages with multiple flavors which is a problem.

Use the Django pkgname prefix for dependencies
Add django-2.0 to the list of possible versions.

ToDo:

fix www/seahub
DJANGO_REL -- release value equivalent to PYTHON_REL

In D12592#283142, @matthew wrote:
Merge upstream changes -- ie. package flavours. www/seahub is
problematic, causing the whole poudriere bulk to fail if included in
the list of package targets. Other than that, builds correctly with
both py27 and py36.

Creating django111 and django20 flavours would be interesting, but
that implies packages with multiple flavors which is a problem.

Use the Django pkgname prefix for dependencies

Add django-2.0 to the list of possible versions.

ToDo:
fix www/seahub
DJANGO_REL -- release value equivalent to PYTHON_REL

It maybe time to rename the port to py-seahub. Is this the issue you are running into? Seahub only runs on py27 and django 18. Let me know if there is anything I can assist with.

In D12592#283656, @ultima wrote:
In D12592#283142, @matthew wrote:
Merge upstream changes -- ie. package flavours. www/seahub is
problematic, causing the whole poudriere bulk to fail if included in
the list of package targets. Other than that, builds correctly with
both py27 and py36.

Creating django111 and django20 flavours would be interesting, but
that implies packages with multiple flavors which is a problem.

Use the Django pkgname prefix for dependencies

Add django-2.0 to the list of possible versions.

ToDo:
fix www/seahub
DJANGO_REL -- release value equivalent to PYTHON_REL
It maybe time to rename the port to py-seahub. Is this the issue you are running into? Seahub only runs on py27 and django 18. Let me know if there is anything I can assist with.

Well, given the dependencies on both Django and Python, by rights that should be www/py-django-seahub. That however is not precisely the issue. www/seahub builds fine using 'poudriere testbuild' but not when using 'poudriere bulk' -- this is with python=2.7 and django=1.8 in DEFAULT_VERSIONS. It's not that the www/seahub port fails under 'poudriere bulk'. The entire poudriere run aborts without building anything like so:

lucid-nonsense:...work/github/freebsd-ports:# poudriere bulk -j 11_1a -z development -p github -f ~matthew/work/django-port-list
[00:00:00] Creating the reference jail... done
[00:00:00] Mounting system devices for 11_1a-github-development
[00:00:00] Mounting ports/packages/distfiles
[00:00:00] Stashing existing package repository
[00:00:00] Mounting ccache from: /var/cache/ccache
[00:00:00] Mounting packages from: /usr/local/poudriere/data/packages/11_1a-github-development
[00:00:00] Copying /var/db/ports from: /usr/local/etc/poudriere.d/development-options
[00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/development-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/11_1a-github-development/ref/etc/resolv.conf
[00:00:00] Starting jail 11_1a-github-development
[00:00:00] Logs: /usr/local/poudriere/data/logs/bulk/11_1a-github-development/2017-12-21_10h19m34s
[00:00:00] Loading MOVED
[00:00:00] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:00] Gathering ports metadata
[00:00:11] Calculating ports order and dependencies
[00:00:11] Error: compute_deps_pkg failed to lookup pkgname for www/py-django@ processing package seahub-6.2.4_1 from www/seahub -- Does www/py-django provide the '' FLAVOR?
[00:00:12] Error: Fatal errors encountered calculating dependencies
[00:00:12] Cleaning up
11_1a-github-development: removed
11_1a-github-development-n: removed
[00:00:12] Unmounting file systems

In D12592#283852, @matthew wrote:
[00:00:11] Error: compute_deps_pkg failed to lookup pkgname for www/py-django@ processing package seahub-6.2.4_1 from www/seahub -- Does www/py-django provide the '' FLAVOR?

This means that www/seahub does not have any flavors, so the RUN_DEPENDS in django.mk ends with ...py-django@ which is interpreted as an empty string flavor. You probably need to replace this with PY_FLAVOR.

In D12592#283856, @mat wrote:

This means that www/seahub does not have any flavors, so the RUN_DEPENDS in django.mk ends with ...py-django@ which is interpreted as an empty string flavor. You probably need to replace this with PY_FLAVOR.

Ahah! Yes, that solves the problem.

Major rework of the code, modelled on python.mk

FLAVOR support
Now requires python.mk to preceed djano.mk in the USES definition
Add a script 'version-rel.sh' to turn typical multipart version numbers into sortable integers
Add Makefile.version to each of the Django ports

This mostly works, but is not yet quite ready for primetime -- test
building all of the Django related ports results with all the
different flavors generates about 700 out of 800 ports correctly.

Fix remaining build problems. Every Django related port now builds
correctly.

Fixes here involve redefining a number of python related variables
using the Python versions determined from the flavoring. This seems
inelegant -- in a couple of places it's literally cut'n'paste from
python.mk -- but it works!

matthew marked 3 inline comments as done.Dec 31 2017, 9:26 PM

matthew edited the summary of this revision. (Show Details)Jan 1 2018, 8:26 AM

matthew marked an inline comment as done.Jan 1 2018, 8:28 AM

Merge branch 'master' into arcpatch-D12592

Merge branch 'master' into arcpatch-D12592

mat added inline comments.Jan 2 2018, 2:38 PM

Mk/Uses/django.mk
203	I do not see this FLAVORS variable defined beforehand, where does it come from exactly ?

matthew added inline comments.Jan 2 2018, 4:14 PM

Mk/Uses/django.mk
203	That's what is set in python.mk

Merge branch 'master' into arcpatch-D12592

mat added inline comments.Jan 3 2018, 1:08 PM

Mk/Uses/django.mk
203	Mmm, but django < python, you can absolutely assume that the sort police will put django before python and it will break. You need to add a `.include "${USESDIR}/python.mk"` before you use any variables coming from python.mk.

matthew added inline comments.Jan 3 2018, 6:59 PM

Mk/Uses/django.mk
203	Agreed that there will be some people who sort the USES line, but not for long. That's why I put in the check for `.if !defined(_INCLUDE_USES_PYTHON_MK)` at line 70 -- all they'll get is an error message which, I trust, will help to educate them that sorting things is not always a good idea. Notice that the body of this file is in the `.else` branch of that test, so none of the python variables will be processed unless python.mk has already been included. If I include `${USESDIR}/python.mk` doesn't that break processing any arguments for the python USES?

Merge branch 'master' into arcpatch-D12592
Add new py-django-netfields port

mat added inline comments.Jan 4 2018, 12:27 PM

Mk/Uses/django.mk
203	USES must work the same way whatever their order is. USES are self contained, and many include other because of dependencies. It should not break anything.

matthew added inline comments.Jan 5 2018, 7:13 AM

Mk/Uses/django.mk
203	That's a problem when you've got several different USES each trying to define FLAVOR and FLAVORS -- it's presumably the last-added USES that wins, which breaks the requirement about invariance to ordering. I'm thinking we need a rule that any variables that a USES .mk file defines should have a distinct namespace eg. PY_FLAVOR, PY_FLAVORS; DJANGO_FLAVOR, DJANGO_FLAVORS. For flavour handling specifically, the USES could append their namespace prefix to a generic variable, say FLAVOR_NAMESPACES+=PY and then a separate block of code in bsd.ports.mk to combine the flavors from the different namespaces (sorted alphabetically), with a mechanism to filter out incompatible combinations (eg. py27 doesn't mix with django20). Individual ports could override this mechanism by setting the un-namespaced variables FLAVOR or FLAVORS as now.

Wouldn't it be easier to have USE_PYTHON+= django:somearg ?

Yeah. This approach is a non-starter. Adding django bits to python.mk is probably a better route as Antoine suggests,
but that'sbasically a start-over.

leres added a subscriber: leres.Mar 9 2019, 5:59 PM

Merge branch 'master' into arcpatch-D12592
Remove leftovers from incomplete merging
Merge branch 'master' into arcpatch-D12592
Remove the py-dj21-django-foo port variants, as these will now be
Merge branch 'master' into arcpatch-D12592
databases/py-carbon no loger has any django dependencies.
Update Makefile.version in each of the main django ports.
Survey the tree to identify all of the Django related ports.
USE_PYTHON=django:1.11+ is identical to the default USES_PYTHON=django
Merge branch 'master' into arcpatch-D12592
Merge branch 'master' into arcpatch-D12592
Django-flavourize newly added www/py-django-cron port
Fix a mis-merge. Mk/Scripts/version-rel.sh was removed from the
DJango-2.1 now in the tree
Reduce the diff to the master branch
Merge branch 'master' into arcpatch-D12592
py-dj21-djangoql -> py-djangoql, flavoured for django 2.1+
Merge branch 'master' into arcpatch-D12592
Merge branch 'master' into arcpatch-D12592
The code at line 299 of Mk/Uses/python.mk already parses an extra
Merge branch 'master' into arcpatch-D12592
New upstream Django 2.2 -- py-django22 port added.
Merge branch 'master' into arcpatch-D12592
Start by defining the outputs -- exported variables and added dependencies.
Merge branch 'master' into arcpatch-D12592
Two more Django ports
Another pass through all the Django ports, switching to the
Generate the django version as a 5 digit integer
Merge branch 'master' into arcpatch-D12592
Merge branch 'master' into arcpatch-D12592
Django 1.8 was dropped from the ports.
Merge branch 'master' into arcpatch-D12592
Merge branch 'master' into arcpatch-D12592
py-logan needs a TEST_DEPENDS on django
Delete extraneous blank line
Generate a 3 digit number from the version for comparison purposes.
Merge branch 'master' into arcpatch-D12592
Merge branch 'master' into arcpatch-D12592
Check that the specified Django version fits within any version
Merge branch 'master' into arcpatch-D12592
Regenerate FLAVORS adding Django version information.
Remove the separate django.mk now that the functionality has been
Add missing '.'
Initialise DJANGO_PORTSDIR before referencing it.
Merge branch 'master' into arcpatch-D12592
Add django to the list of valid categories
Revert the approach of using PY_FLAVOR instead of FLAVOR -- FLAVOR is
Merge branch 'master' into arcpatch-D12592
Have only one USE_PYTHON line for best results
Multiple debug fixes
Adjust CONFLICTS_INSTALL
PyPi claims that the maximum supported branch of Django for
Merge branch 'master' into arcpatch-D12592
Despite the name, django-modelcluster has no dependency on django, so
Where django is only used as a test dependency, the FLAVOR settings
Whitespace
Merge branch 'master' into arcpatch-D12592
With USE_PYTHON=django, the string 'django' will always appear in
Merge branch 'master' into arcpatch-D12592
Set FLAVORS immediately.
Merge branch 'master' into arcpatch-D12592
Merge branch 'master' into arcpatch-D12592
Merge branch 'master' into arcpatch-D12592

This is very much a work in progress and not at all useable yet.

Harbormaster completed remote builds in B29065: Diff 67553.Jan 31 2020, 4:00 PM

matthew added a reviewer: kai.Jan 31 2020, 4:01 PM

Django 2.1 is gone, django 3.0 is here.

Harbormaster completed remote builds in B29067: Diff 67557.Jan 31 2020, 4:13 PM

Merge py-django-rq and py-dj22-django-rq
Tidy up cases where django is bought in as a test dependency
Don't support nodjangoflavor as an option any more.

Harbormaster completed remote builds in B29068: Diff 67559.Jan 31 2020, 4:38 PM

mat added inline comments.Feb 1 2020, 10:25 AM

Mk/Uses/python.mk
276 ↗	(On Diff #67559)	This says SUFFIX, should be PREFIX.
313–317 ↗	(On Diff #67559)	Can you try to only change what you need, adding whitespace and rewrapping differently is not great.
702 ↗	(On Diff #67559)	By definition, the default flavor is the first of the flavors. . if empty(FLAVOR) FLAVOR:= ${FLAVORS:[1]} . endif

matthew marked 2 inline comments as done.Feb 3 2020, 9:25 AM

matthew added inline comments.

Mk/Uses/python.mk
276 ↗	(On Diff #67559)	This simply parallels the comment on PYTHON_PKGNAMEPREFIX at lines 242-245: # PYTHON_PKGNAMEPREFIX # - Use this as a ${PKGNAMEPREFIX} to distinguish # packages for different Python versions. # default: py${PYTHON_SUFFIX}- `${PYTHON_SUFFIX}` is maybe a badly named variable, but changing it would be a lot of churn.

Don't re-wrap _VALID_PYTHON_FEATURES
The default flavor is the first of the flavours.
Merge branch 'master' into arcpatch-D12592

Harbormaster completed remote builds in B29121: Diff 67684.Feb 3 2020, 9:32 AM

Diff against master

Harbormaster completed remote builds in B29122: Diff 67685.Feb 3 2020, 9:35 AM

Merge branch 'master' into arcpatch-D12592

Harbormaster completed remote builds in B29249: Diff 67993.Feb 9 2020, 11:30 AM

Merge pull request #1 from knobix/arcpatch-D12592

Harbormaster completed remote builds in B29635: Diff 68869.Feb 27 2020, 6:58 AM

While there has been a comment on the length of DJANGO_PKGNAMEPREFIX no one seems to have suggested shortening it.

py37-django30-portname

could be shortened to

py37-dj30-portname

port folder names could be similar - py-dj-portname

In D12592#524776, @FreeBSD_ShaneWare.Biz wrote:

While there has been a comment on the length of DJANGO_PKGNAMEPREFIX no one seems to have suggested shortening it.

py37-django30-portname

could be shortened to

py37-dj30-portname

This is certainly something to consider -- many Django modules already have 'django' in their name, so having it twice in the package name is pretty redundant.

port folder names could be similar - py-dj-portname

However, I don't see much value in this. As I said, most Django-related ports already self-identify as such. This patch already makes hundreds of changes to the tree, and I don't think adding quite a lot of additional churn on top of that, to no real benefit, would be a good idea.

kai mentioned this in D24850: www/py-djangorestframework: Update to 3.11.0.May 21 2020, 1:57 PM

dbaio added a subscriber: dbaio.Oct 20 2020, 2:43 AM

swills added a subscriber: swills.Oct 20 2020, 8:05 PM

sbz resigned from this revision.Apr 19 2021, 5:13 PM

This is well out of date.

New USES+=django and Django default versionAbandonedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 33860

Mk/Uses/django.mk

Mk/bsd.default-versions.mk

databases/py-carbon/Makefile

databases/py-django-transaction-hooks/Makefile

devel/py-django-rq-dashboard/Makefile

devel/py-django-rq/Makefile

devel/py-django-tastypie-mongoengine/Makefile

devel/py-logan/Makefile

graphics/py-django-easy-thumbnails/Makefile

graphics/py-sorl-thumbnail/Makefile

security/py-first-server/Makefile

textproc/pootle/Makefile

www/feedjack/Makefile

www/py-django-allauth/Makefile

www/py-django-annotations/Makefile

www/py-django-annoying/Makefile

www/py-django-appconf/Makefile

www/py-django-appmedia/Makefile

www/py-django-assets/Makefile

www/py-django-auth-ldap/Makefile

www/py-django-babel/Makefile

www/py-django-bitfield/Makefile

www/py-django-bootstrap-form/Makefile

www/py-django-bulk-update/Makefile

www/py-django-caching-app-plugins/Makefile

www/py-django-classy-tags/Makefile

www/py-django-cms/Makefile

www/py-django-configurations/Makefile

www/py-django-constance/Makefile

www/py-django-contact-form/Makefile

www/py-django-contrib-comments/Makefile

www/py-django-cors-headers/Makefile

www/py-django-cors-headers/distinfo

www/py-django-countries/Makefile

www/py-django-crispy-forms/Makefile

www/py-django-datetime-widget/Makefile

www/py-django-debug-toolbar/Makefile

www/py-django-dpaste/Makefile

www/py-django-evolution/Makefile

www/py-django-extensions/Makefile

www/py-django-filer/Makefile

www/py-django-formtools/Makefile

www/py-django-guardian/Makefile

www/py-django-happenings/Makefile

www/py-django-haystack/Makefile

www/py-django-hijack/Makefile

www/py-django-htmlmin/Makefile

www/py-django-json-rpc/Makefile

www/py-django-jsonfield/Makefile

www/py-django-jsonfield/distinfo

www/py-django-keyedcache/Makefile

www/py-django-ldapdb/Makefile

www/py-django-livesettings/Makefile

www/py-django-markdownx/Makefile

www/py-django-markwhat/Makefile

www/py-django-mezzanine-filebrowser/Makefile

www/py-django-mezzanine-grappelli/Makefile

www/py-django-mezzanine/Makefile

www/py-django-mptt/Makefile

www/py-django-otp-yubikey/Makefile

www/py-django-otp/Makefile

www/py-django-overextends/Makefile

www/py-django-paging/Makefile

www/py-django-photologue/Makefile

www/py-django-picklefield/Makefile

www/py-django-pipeline/Makefile

www/py-django-post_office/Makefile

www/py-django-profiles/Makefile

www/py-django-pyscss/Makefile

www/py-django-recaptcha/Makefile

www/py-django-redis/Makefile

www/py-django-registration-defaults/Makefile

www/py-django-registration-redux/Makefile

www/py-django-registration/Makefile

New USES+=django and Django default version
AbandonedPublic
Actions

Revision Contents
Changeset List