lang/go: Provide only one go version

[adamw]

Summary

The ultimate goal here is to drop down to providing one version of Go.

It should essentially be a NOOP; depending on the go value in go.mod, Go will already download and add packages for a newer version, and will restrict itself to features present in an older version.

There's really no value anymore in providing old versions of Go. Someday a go-devel could get created, but that's essentially going to be it.

I'm starting with a patch that just forces 1.24 as the version for everything. If things look fine, then I'll rip out go.mk's guts and do some smaller follow-up testing.

[adamw]

@danilo are you able to please test build all the go ports?

[kbowling]

Is this really feasible? At first contact I would think this will create a lot of different problems than whatever supporting multiple versions currently does. Some projects have elaborate dependencies (i.e. multimedia/navidrome which I maintain) and we could get into common situations where it is either difficult to update go with a clean exp-run or we have to make a choice to regularly lose ports because tracking a modern version is more important.

[adamw]

In D49906#1137925, @kbowling wrote:

Is this really feasible? At first contact I would think this will create a lot of different problems than whatever supporting multiple versions currently does. Some projects have elaborate dependencies (i.e. multimedia/navidrome which I maintain) and we could get into common situations where it is either difficult to update go with a clean exp-run or we have to make a choice to regularly lose ports because tracking a modern version is more important.

The point is that Go now builds the same regardless of which version you use. If the go.mod says to use a newer version, it builds the stdlib from that version and links it in. And if go.mod says to use an older version, go restricts its feature set to behave exactly as that old version would.

So, whether we have one Go version in the tree or ten, the output will be exactly the same. There's literally no difference between Go versions anymore, so we gain nothing by having multiple in the tree.

[kbowling]

In D49906#1137930, @adamw wrote:

In D49906#1137925, @kbowling wrote:

Is this really feasible? At first contact I would think this will create a lot of different problems than whatever supporting multiple versions currently does. Some projects have elaborate dependencies (i.e. multimedia/navidrome which I maintain) and we could get into common situations where it is either difficult to update go with a clean exp-run or we have to make a choice to regularly lose ports because tracking a modern version is more important.

The point is that Go now builds the same regardless of which version you use. If the go.mod says to use a newer version, it builds the stdlib from that version and links it in. And if go.mod says to use an older version, go restricts its feature set to behave exactly as that old version would.

So, whether we have one Go version in the tree or ten, the output will be exactly the same. There's literally no difference between Go versions anymore, so we gain nothing by having multiple in the tree.

Ok, I'm tracking on the backward compat. Although I am skeptical about the last sentence, i.e. we will need an equal or newer compiler to handle all go.mod versions. But that is a tractable situation. And exp-runs will shake out anything that is not adequately pinned.

[adamw]

In D49906#1138150, @kbowling wrote:

In D49906#1137930, @adamw wrote:

There's literally no difference between Go versions anymore, so we gain nothing by having multiple in the tree.

Ok, I'm tracking on the backward compat. Although I am skeptical about the last sentence, i.e. we will need an equal or newer compiler to handle all go.mod versions.

I know... I have some skepticism too, though this is how Go says it's supposed to work. I'm curious to see what that exp-run looks like too!

But: new shouldn't matter either (in theory), because if you have 1.22 and go.mod says 1.24, Go will download, compile, and link in the newer toolchain.

Go definitely behaving in a well-reasoned and polite manner, but it's just completely contrary to how the ports tree expects builders to behave.

[dbaio]

@adamw I've just opened an exp-run request on 286214.

[fuz]

I disagree with this step.

We should instead disable the feature where Go tried to download a toolchain and have Go err out instead. This can be achieved by setting GOTOOLCHAIN=local in the environment. Packages should always be built with our own toolchain, not the one provided by Google. The downloaded toolchain has none of our site patches, is outside of our control and we don't do this sort of thing anywhere else.

Please also consider that there are ports which don't build with a newer Go version. While rare, this occasionally happens and may require using an older than the default toolchain.

[adamw]

In D49906#1138522, @fuz wrote:

I disagree with this step.

We should instead disable the feature where Go tried to download a toolchain and have Go err out instead. This can be achieved by setting GOTOOLCHAIN=local in the environment. Packages should always be built with our own toolchain, not the one provided by Google. The downloaded toolchain has none of our site patches, is outside of our control and we don't do this sort of thing anywhere else.

Everything you said is true, but that's just not how Go is designed to work anymore, and I'm all for respecting that. There's literally no difference between go124 building a go123 program, and go123 building a go123 program. You're talking about a ton of extra complexity just to be able to say that we built it with a different version. The output should be identical (should but who knows).

Please also consider that there are ports which don't build with a newer Go version. While rare, this occasionally happens and may require using an older than the default toolchain.

You're missing the point, I suspect. Go no longer has that problem. Go will no longer try to build go121 programs with a go124 stdlib. It will build go121 with a feature set identical go121, and this happens transparently and automatically. Go designed it this way so that there is never a need for multiple go versions. Technically we could leave go122 in the ports tree and it'd build all future Go programs.

So I mean, you're not wrong on anything, and what you're proposing is definitely how we expect the ports tree to work. My argument here is that that's not how Go is designed to work anymore, and I'd rather make the ports tree leverage Go, rather than using the ports tree to dismantle and override Go.

And as for local patches: the patches we apply control the build process, which is always handled by the port itself. So, as far as I can tell, we're not losing any of our patches here (though please correct me if I'm wrong!).

[adamw]

So, I should have tagged this review in the commit, but I set go124 as the default as portmgr/pkgmgr seems to be swamped at the moment. Once we see how things build, we'll have a better picture of whether this will be a viable thing.

[fuz]

You're missing the point, I suspect. Go no longer has that problem. Go will no longer try to build go121 programs with a go124 stdlib. It will build go121 with a feature set identical go121, and this happens transparently and automatically. Go designed it this way so that there is never a need for multiple go versions. Technically we could leave go122 in the ports tree and it'd build all future Go programs.

So, I should have tagged this review in the commit, but I set go124 as the default as portmgr/pkgmgr seems to be swamped at the moment. Once we see how things build, we'll have a better picture of whether this will be a viable thing.

If the test build shows that none of the ports fail with a more recent toolchain, I'm okay with only providing it. But what we must disable in the ports tree is the mechanism where Go tries to download a newer toolchain if it considers itself outdated. Downloading random toolchains at fetch time is not what we should be doing.

And as for local patches: the patches we apply control the build process, which is always handled by the port itself. So, as far as I can tell, we're not losing any of our patches here (though please correct me if I'm wrong!).

We used to have patches in the past that affected the binaries built, such as to work around some arm shenanigans. Are you confident that we will never need such patches in the future? Because downloading toolchains at fetch time means that we lose the ability to patch them.

[adamw]

In D49906#1151683, @fuz wrote:

If the test build shows that none of the ports fail with a more recent toolchain, I'm okay with only providing it. But what we must disable in the ports tree is the mechanism where Go tries to download a newer toolchain if it considers itself outdated. Downloading random toolchains at fetch time is not what we should be doing.

I love that behaviour in general, but I agree that it's not a great fit for the ports tree.

But the easiest way to prevent that is simply to ensure that we keep providing the latest go release. As long as the minor doesn't exceed what's in the ports tree, that toolchain-downloading behaviour will never trigger.

And as for local patches: the patches we apply control the build process, which is always handled by the port itself. So, as far as I can tell, we're not losing any of our patches here (though please correct me if I'm wrong!).

We used to have patches in the past that affected the binaries built, such as to work around some arm shenanigans. Are you confident that we will never need such patches in the future? Because downloading toolchains at fetch time means that we lose the ability to patch them.

No, I have absolutely no confidence there. What I am confident about is that if we keep Go updated then (at least on paper) it should be irrelevant. But there are definitely some big Ifs there.

[fuz]

But the easiest way to prevent that is simply to ensure that we keep providing the latest go release. As long as the minor doesn't exceed what's in the ports tree, that toolchain-downloading behaviour will never trigger.

The best way would be to set GOTOOLCHAIN=local when building Go ports, which disables the “fetch the appropriate toolchain” mechanism. This makes it impossible to build Go ports requiring too new toolchains, but then perhaps there's more of an incentive to have them packaged quickly.

[adamw]

In D49906#1152227, @fuz wrote:

The best way would be to set GOTOOLCHAIN=local when building Go ports, which disables the “fetch the appropriate toolchain” mechanism. This makes it impossible to build Go ports requiring too new toolchains, but then perhaps there's more of an incentive to have them packaged quickly.

Totally. OTOH, in addition to incentive, it also provides us some flexibility in getting updates rolled out. If it takes us a month to get an update applied, there's a high chance that nearly all ports will continue building, even if they specify Go 1.n+1. I'm not 100% clear that that's a good thing in the end, but it definitely could give us some wiggle room.

[kbowling]

Is there a way to vendor the alt toolchains? It is imperative this doesn't start downloading from the internet outside of the fetch step, the same standards things like Cargo, Maven, etc have in ports. As long as that is addressed I think the overall concept is fine now -- it will be awkward if we have to patch Go but hopefully that is not common.

[adamw]

In D49906#1153054, @kbowling wrote:

Is there a way to vendor the alt toolchains? It is imperative this doesn't start downloading from the internet outside of the fetch step, the same standards things like Cargo, Maven, etc have in ports. As long as that is addressed I think the overall concept is fine now -- it will be awkward if we have to patch Go but hopefully that is not common.

As far as Go is concerned, a newer toolchain is just another package, known in advance like any other. The compiler shouldn't have any need to grab anything outside of fetch. This behavior has been live in the tree for a while now without poudriere filesystem access errors.