HomeFreeBSD

ports-mgmt/portscout: Add GitHub and PyPI site handlers & MORE!

Description

ports-mgmt/portscout: Add GitHub and PyPI site handlers & MORE!

Over the past several months portscout.freebsd.org appears to have been
more frequently NOT finding updates, particularly for ports that use
CHEESESHOP (PyPI) as their MASTER_SITES.

Portscout has also never worked for ports using GitHub for distribution
files due to the following:

a) Portscout, prior to 'guessing', requests a randomly named file
   from the Site and expects a 4xx (404) in response. If it doesn't
   receive a 4xx response, it increments a 'lie counter' and does not
   check the site again in the next run.
b) The GitHUB handlers (SUBDIR/MASTER_SITES) in bsd.sites.mk
   construct a URL that ends in a a dummy query paramater (for the
   filename), so that fetch saves the correct filename to DISTDIR.
   This means for any DISTFILE name provided, a 200 OK response is
   returned

These two factors unfortunately leave us in a position where there is no
good way to workaround this in the ports framework, including overriding
DISTFILES, DISTNAME, FETCH_ARGS, or the SUBDIR URL itself for various
reasons (not matching distinfo, file conflicts in DISTDIR, etc)

Fortunately, the portroach project (OpenBSD's fork of portscout)
contains a site handler for GitHub and PyPI (among others) already [1].

These site handlers use API endpoints at GitHub and PyPI that respond
JSON respectively, providing a faster and more accurate way to determine
the latest version of a package, without having to go through the
'guessing' process.

This commit:

  • Adds GitHub and PyPI site handlers, and modifies or extends them to accept/match our MASTER_SITES URL's.
  • Adds authenticated API request support and two settings for the GitHub site handler
  • Add p5-JSON to RUN_DEPENDS (needed by new site handlers)
  • Add HTTPS option for supporting https:// MASTER_SITES. Currently portscout does not check (fails) https:// MASTER_SITES [2]
  • Take MAINTAINER'ship
  • Adds badly needed logging/debugging messages to key parts of the process retaining the conditional logic that ties the verbosity to "quiet" or "debug" portscout.conf settings.
  • Renables the SQLITE3 option (previously commented out) and renames it to SQLITE (the standard, as per bsd.options.desk.mk)
  • Creates a DATABASE option group allowing either/or SQLITE or POSTGRESQL to be selected
  • Switches option conditionals where possible to options helpers
  • Backport a fix for maintainer matching/mapping [3]
  • Adds LICENSE (BSD2CLAUSE)
  • Updates and sorts pkg-plist

This change was tested again ports maintained by me, and resulted in
'new versions' being found and reported for 42 of my ports (of 123).

[1] https://github.com/jasperla/portroach/tree/master/Portroach/SiteHandler
[2] Reported by: truckman
[3] https://github.com/jasperla/portroach/commit/2f6ee134ddc178f74688b37b986b66ecb0481782

PR: 203996
Approved by: maintainer (timeout, 15 days)
Differential Revision: https://reviews.freebsd.org/D3996

Details

Provenance
koobsAuthored on
Differential Revision
D3996: ports-mgmt/portscout: Add GitHub and PyPI site handlers
Parents
rP401036: - Update to 1.2.1
Branches
Unknown
Tags
Unknown