Page MenuHomeFreeBSD

Mk/bsd.sites.mk should use HTTPS whenever possible
ClosedPublic

Authored by wosch on Feb 4 2018, 5:30 PM.
Tags
None
Referenced Files
Unknown Object (File)
Feb 18 2024, 8:16 AM
Unknown Object (File)
Feb 12 2024, 4:28 AM
Unknown Object (File)
Dec 21 2023, 7:13 PM
Unknown Object (File)
Dec 20 2023, 6:48 AM
Unknown Object (File)
Dec 10 2023, 5:45 AM
Unknown Object (File)
Dec 4 2023, 12:51 AM
Unknown Object (File)
Dec 4 2023, 12:22 AM
Unknown Object (File)
Oct 14 2023, 5:55 PM

Details

Summary

The file Mk/bsd.sites.mk contains a list of mirror sites for software repos. Many of the sites supports HTTPS, but we are still using HTTP.

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Just a note, that does not improve anything security wise, given all the distfiles avec validated against checksums before getting used. that said, I'm not against that change :)

In D14194#297693, @bapt wrote:

Just a note, that does not improve anything security wise, given all the distfiles avec validated against checksums before getting used. that said, I'm not against that change :)

The issue is about privacy, not security. To quote wikipedia: "The domain of privacy partially overlaps security (confidentiality), which can include the concepts of appropriate use, as well as protection of information."

While I agree with bapt I think we should offer HTTPS where possible as this seems to be a concern for some people and many sites will redirect to HTTPS anyway.
That said, it wouldn't hurt to give Mk/bsd.sites.mk an overhaul in general.

I've done similar work for other open source distros and can give it a go using the following guidelines if there's interest.
I'd like to do it per master site and not do a massive diff.

  • 8-10 (mirror)-sites in total, fall back to main site as last resort
  • HTTPS (where possible) --> HTTP --> FTP (in that order)
  • Prioritize fast mirrors in general with good connectivity (peering/backbone) for NA/EU users however provide at least 2 mirrors in Asia for connectivity and availability if possible
In D14194#297712, @daniel.engberg.lists_pyret.net wrote:
  • 8-10 (mirror)-sites in total, fall back to main site as last resort
  • HTTPS (where possible) --> HTTP --> FTP (in that order)
  • Prioritize fast mirrors in general with good connectivity (peering/backbone) for NA/EU users however provide at least 2 mirrors in Asia for connectivity and availability if possible

I am thrilled with points 2. Point 2 would be HUGE for me.

I'm not sure that there's much benefit to limiting the number of mirrors. There's no reason to remove mirrors unless they don't work, or they're prohibitively slow. I'm fine with prioritizing NA and EU sites, but there's no reason to remove Asian mirrors.

There's no point in having like 15+ mirrors, in general I'd say if 4+ fails there's another general issue than the mirrors themselves and it'll just take ages before you hit FreeBSD's own distfiles servers.
What I meant is that Asian mirrors shouldn't be the first choice as they're going to be slow for NA/EU users in general which is the majority of users (?) but still useful when it comes to connectivity and available hence why they should be included.
Given what I've mentioned earlier there's no need for a boatload of listed ones, it'll just be a hassle in the end.

I have no intention of down prioritize Asian users but unfortunately there's still quite a bit of latency for most users accessing sites which isn't ideal.

Not all mirrors update at the same frequency. It's not uncommon (when following head) to go 10 mirrors down before one fetches.

I would rather have 4 of 20 mirrors fetch than 0 of 5.

I think a good place to start is https > http > ftp. I'd prefer to see https added at higher priority in each master site, than to see http replaced altogether (as this review does). I'm fine with reducing the number of http in each mirror (replacing them with https), but removing all http mirrors from a group is a disservice to a segment of our user base.

One problem with a change like this is that it prevents most proxies from being able to cache the files for 'privacy'. It's still obvious a connection is made to these mirrors, just not which file is being fetched. Does that really matter?

While it might be an potential issue I'm not sure it is in the end as most of these sites hosts files several megabytes in size which most proxies doesn't cache anyway?

One problem with a change like this is that it prevents most proxies from being able to cache the files for 'privacy'. It's still obvious a connection is made to these mirrors, just not which file is being fetched. Does that really matter?

I think that caching becomes less important now. Network traffic is damn cheap (<1 Euro per TB). If you need to reduce the network traffic or latency for your FreeBSD machines, you may setup your own distcache server.

In D14194#297749, @daniel.engberg.lists_pyret.net wrote:

There's no point in having like 15+ mirrors

Yes there is. To allow our users to choose which they want first, using the MASTER_SORT feature.

In D14194#298115, @mat wrote:
In D14194#297749, @daniel.engberg.lists_pyret.net wrote:

There's no point in having like 15+ mirrors

Yes there is. To allow our users to choose which they want first, using the MASTER_SORT feature.

I didn't know that existed at all so thanks for bringing that up however in all fairness many sites these days aren't easily geographically identifiable by domain and/or sub domains so it may only be useful to some extent.
Worth nothing is that back in August bapt did a remove a lot of dead hosts which leads me to believe that having lot of hosts that rarely or never gets used only adds to more potential dead hosts that "never" gets noticed. I highly doubt that all were closed down closely in time to each other in the first place. I do think however that mirrors should be unique which would being more diversity comped to just switching protocol for the same host and should bring enough of choice for most users?

About the proxy caching, yes it is true that it might break but as it has been stated those files are anyway usually to big to be actually cached.
As for the privacy brought by https, it is quite easy to guess which distfiles are being downloaded base on the size.
And about fast network access, there are still plenty of countries/places without fast and/or cheap internet access.
Also in the case none of the above is relevant, so let's go forward and have this in :)

This revision is now accepted and ready to land.Feb 6 2018, 10:32 AM
This revision was automatically updated to reflect the committed changes.

I've seen this too late (the change was already committed), but for the record I'm also against HTTPS for master sites (HTTPS is generally a huge pain for me, and I tend to avoid it when it is not required). Oh well, I guess I can maintain a local patch to bsd.sites.mk (with this and other similar revisions reverted).

Perhaps it would make sense to make preferred order (http[s]/ftp/etc.) configurable?