ports-mgmt/portscout: Add GitHub and PyPI site handlers

[koobs]

Summary

Over the past several months portscout.freebsd.org appears to have been
more frequently NOT finding updates, particularly for ports that use
CHEESESHOP (PyPI) as their MASTER_SITES.

Portscout has also never worked for ports using GitHub for distribution
files due to the following:

• Portscout, prior to 'guessing', requests a randomly named file from the Site and expects a 4xx (404) in response. If it doesn't receive a 4xx response, it increments a 'lie counter' and does not check the site again in the next run.
• The GitHUB handlers (SUBDIR/MASTER_SITES) in bsd.sites.mk construct a URL that ends in a a dummy query paramater (for the filename), so that fetch saves the correct filename to DISTDIR. This means for any DISTFILE name provided, a 200 OK response is returned

These two factors unfortunately put us in a position where there is no
good way to workaround this in the ports framework, including overriding
DISTFILES, DISTNAME, FETCH_ARGS, or the SUBDIR URL itself for various
reasons (not matching distinfo, file conflicts in DISTDIR, etc)

Fortunately, the portroach project (OpenBSD's fork of portscout)
contains a site handler for GitHub and PyPI (among others) already [1].

These site handlers use API endpoints at GitHub and PyPI that respond
JSON respectively, providing a faster and more accurate way to determine
the latest version of a package, without having to go through the
'guessing' process.

This commit:

• Adds GitHub and PyPI site handlers, and modifies or extends them to accept/match our MASTER_SITES URL's.
• Adds authenticated API request support and two settings for the GitHub site handler
• Add p5-JSON to RUN_DEPENDS (needed by new site handlers)
• Add HTTPS option for supporting https:// MASTER_SITES out of the box
• Take MAINTAINER'ship
• Adds badly needed logging/debugging messages to key parts of the process retaining the conditional logic that ties the verbosity to "quiet" or "debug" portscout.conf settings.
• Renables the SQLITE3 option (previously commented out) and renames it to SQLITE (the standard, as per bsd.options.desk.mk)
• Creates a DATABASE option group allowing either/or SQLITE or POSTGRESQL to be selected
• Switches option conditionals where possible to options helpers
• Backport a fix for maintainer matching/mapping [2]
• Adds LICENSE (BSD2CLAUSE)
• Updates and sorts pkg-plist

This change was tested again ports maintained by me, and resulted in
'new versions' being found and reported for 42 of my ports (of 123).

[1] https://github.com/jasperla/portroach/tree/master/Portroach/SiteHandler
[2] https://github.com/jasperla/portroach/commit/2f6ee134ddc178f74688b37b986b66ecb0481782

Test Plan

• portlint: OK (looks fine)
• testport: OK (poudriere)
• my ports: 42 new versions found (CHEESESHOP, GITHUB) of 123 ports. portscout currently shows none

[koobs]

Update differential with a manual diff created through svn diff because arc diff --create didnt seem to add the contents of the newly added files. cc @eadler

[mat]

It seems peculiar how portscout does things in a strange way, it parses the url, and try to bump the versions in it when it could simply ask the ports infrastructure the url for a newer URL, say:

benchmarks/flops $ make fetch-urlall-list
https://codeload.github.com/AMDmi3/flops/tar.gz/2.1?dummy=/AMDmi3-flops-2.1_GH0.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/AMDmi3-flops-2.1_GH0.tar.gz
benchmarks/flops $ make fetch-urlall-list PORTVERSION=2.2
https://codeload.github.com/AMDmi3/flops/tar.gz/2.2?dummy=/AMDmi3-flops-2.2_GH0.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/AMDmi3-flops-2.2_GH0.tar.gz

[koobs]

• Fix divide by zero error caused by trying to print strings (in variables) outside of quotes.
• Remove new skip_liars setting for now (for future patch)
• Fix unintended reverse logic settings{debug} -> settings{quiet}
• Fix 2 x unintended reverse logic "unless debug" -> 'if debug'
• Dont show "Matched cat/port" messages 'if quiet'
• Dont show 'Site handler' messages 'if quiet'
• Simplify and polish site handler log messaging

[koobs]

Update diff manually again because new file contents are not being added/sent to phabricator via arc cc @eadler

[koobs]

• Add authenticated API request support to the GitHub site handler
• Document the new options in portscout.conf

This allows the client to make requests at a much higher rate (total: 5000/hr) instead of 60 requests/hr in the case of unauthenticated (anonymous) requests. We'll need this for portscout.fo.o

[koobs]

Take MAINTAINER'ship

[truckman]

ports-mgmt/portscout/Makefile

23 ↗

(On Diff #9694)

In order for portscount to work with https MASTER_SITES, it either needs to include p5-LWP-Protocol-https>=0:${PORTSDIR}/www/p5-LWP-Protocol-https in RUN_DEPENDS, or p5-libwww needs to be built with the HTTPS option enabled, which is not the default.

[koobs]

ports-mgmt/portscout/Makefile

23 ↗

(On Diff #9694)

Perfect thanks! I'm glad we isolated what the issue was. I'll add an SSL (HTTP MASTER_SITES) option, enable it by default, and add SSL_RUN_DEPENDS for it.

Better too if we depend on it directly I think

[koobs]

Add HTTPS option (And enable by default)