Mk/Uses/go.mk: Add support for using Go for dependency management

[dmgk]

Summary

Add (experimental) support for offloading dependency management to Go using
go mod download.

Using GH_TUPLE for managing Go port dependencies has several issues:

• Go dependencies are modules, not repositories that GH_TUPLE was designed to work with. A single repo can often contain multiple independently versioned Go modules and this arrangement is hard to work with using GH machinery.

• Because of the above, modules2tuple often has difficulties handling corner cases. This makes porting of some Go software unnecessarily difficult.

• Not all dependencies have Github mirrors.

• Port maintainers sometimes forget to update {GH,GL}_TUPLE dependencies. The port may still build but the resulting binary is not what upstream tested or released.

• Due to Go getting stricter about validating vendored deps in recent versions, we have to use patched Go version to relax these restrictions [1].

On the other hand, Go already know how to download [2] and verify integrity
[3] of all dependencies required for building a Go port. None of the drawbacks
listed above apply and required configuration in some cases can be reduced to
a single variable GO_MODULE.

How to switch:

• Make sure that the port uses Go modules (USES=go:modules)

• Add GO_MODULE=... variable and set it to the module name as shown in the module directive in upstream's go.mod.

• Remove USE_GITHUB/USE_GITLAB and related GH/GL variables (GH_ACCOUNT, GH_PROJECT, GH_TUPLE etc).

• make makesum to update distinfo and populate module cache.

The only upstream used in this mode is an instance of Go module proxy
(https://proxy.golang.org by default). This can be changed by setting GO_GOPROXY
variable.

Example:

• Current devel/goswagger: https://gist.github.com/dmgk/47af523fee37fb524fd87b5ec718844a
• After conversion: https://gist.github.com/dmgk/fb9e314c6229c09e07d0ed5ab5bf55dd

[1] https://github.com/golang/go/issues/37948
[2] https://golang.org/ref/mod#go-mod-download
[3] https://www.goproxy.io/docs/GOSUMDB-env.html

Test Plan

No general test plan because this change is backwards-compatible and ports can
be switched gradually on case-by-case basis.

[dmgk]

Add GO_GOSUMDB variable.

[adamw]

This is a clever approach! If I'm reading it correctly, this should also fix the case of unnecessary (i.e. test/bench) modules being dragged in and not used in the final build.

AFAIK this would be the first instance of distfiles being fetched by anything except fetch(1). What are the consequences of that likely to be? Are there proxy etc. settings that fetch will honor but go mod download won't?

[dmgk]

In D28184#629947, @adamw wrote:

This is a clever approach! If I'm reading it correctly, this should also fix the case of unnecessary (i.e. test/bench) modules being dragged in and not used in the final build.

Hi Adam,

No, unfortunately this change won't make downloads smaller. Go currently is not smart enough to figure out what's really needed for the build and download only required bits.

This change is mostly aimed to ease porting/maintenance and to stop relying on undocumented bits that require us to patch Go (and other drawbacks listed in the summary).

AFAIK this would be the first instance of distfiles being fetched by anything except fetch(1). What are the consequences of that likely to be? Are there proxy etc. settings that fetch will honor but go mod download won't?

With this change, only one upstream/MASTER_SITE is used - the Go modules proxy (https://proxy.golang.org by default, configurable). Everything gets downloaded from it, including the port distfile. Because go uses its stdlib net/http module, it will automatically respect http_proxy settings [1].

[1] https://golang.org/pkg/net/http/#RoundTripper

[dmgk]

In D28184#629947, @adamw wrote:

AFAIK this would be the first instance of distfiles being fetched by anything except fetch(1).

To be clear, DISTFILES are still fetched with fetch(1) and checksummed in the usual way. It is only dependencies that are downloaded with go mod download in post-fetch.

[dmgk]

• Set GOPATH to ${DISTDIR}/go/${PKGORIGIN}
• Hook up cleaning module cache to distclean

[dmgk]

• Clean the whole GOPATH on distclean

[dmgk]

• Flatten ${DISTDIR}/go hierarchy a bit by substituting / with _ in ${PKGORIGIN}
• Vendor dependencies in post-extract. We still need to do this to be able to apply local port patches to dependencies.

[dmgk]

• Do not test for vendor dir existence in post-extract, always vendor. Distfiles served by module proxy do not include vendored deps.

[swills]

It's unfortunate that we have to do this, but I don't see any issue with this change.