FOSDEM: Notes from MirrorBrain talk
Feb. 6th, 2010 06:11 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
robbat2Sitting in the MirrorBrain talk at FOSDEM, taking notes.
Actively used since ~2007.
Split between the redirector and the tester, explicitly made separate.
SourceForge helped with the ASN/Closest-Network side.
Metalinks and P2P support.
Scans mirrors for filelist to see what's present.
Load limiting by making director support mirrors that are limited to a local network / AS / country etc.
MetaLinks don't have Magnet links presently, but I noted that it should be possible to include it.
Using GeoDNS directly for lookups can cause trouble with partial mirrors. Ideally need to put a MirrorBrain server on each continent/region, and GeoDNS to point to that. Also, from some countries, bandwidth to adjcaent countries that might have a mirror is MUCH worse than bandwidth to a well-connected country elsewhere. Past user experience noted with a user in Mozambique, for whom the fastest mirror was via satellite to Canada. Routing data IS needed to make that best choice.
MirrorBrain mailing lists also have a generic non-project-specific "networkers" list for talk between content providers and mirror admins, non-specific to any app.
Comments to your MirrorBrain notes
Date: 2010-02-16 06:19 pm (UTC)thanks for visiting the talk & the discussion!
I saw your notes, and I'd like to add my comments to some things, if you don't mind:
> SourceForge helped with the ASN/Closest-Network side.
It was vice versa :-) I helped them, and they use my code for this purpose. I'm not sure if they also use mod_asn, the lookup Apache module, but they definitely use the database backend that I designed for that.
http://mirrorbrain.org/news/sourceforge_uses_part_of_mirrorbrain/
> MetaLinks don't have Magnet links presently, but I noted that it should be possible to include it.
The one part that's missing here is a better way to store hashes. Right now, the hashes that MirrorBrain stores are stored in a way that would require MirrorBrain to parse a file to construct a magnet link. I'd rather want to avoid that. So, the hash cache needs to undergo a little redesign (which is planned anyway), and then magnet links will be trivial to implement.
> Using GeoDNS directly for lookups can cause trouble with partial mirrors.
Yes, similar as DNSrr causes trouble with partial mirrors.
> Ideally need to put a MirrorBrain server on each continent/region, and GeoDNS to point to that.
That would be my favourite setup for failover and load balancing, and for minimizing latency for the clients (which matters in particular when clients have to request lots of files).
> Also, from some countries, bandwidth to adjcaent countries that might have a mirror is MUCH worse than bandwidth to a well-connected country elsewhere.
... the best link almost always being either to Central Europe, to the U.S., or both.
> Past user experience noted with a user in Mozambique, for whom the fastest mirror was via satellite to Canada.
> Routing data IS needed to make that best choice.
In fact, routing data doesn't help with that. (Or I haven't found a way to make it useful.) Thus, what I use to deal with these cases is manual configuration. In effect, I configure an association of client countries to mirrors where the connection is known to work (always under the condition no local mirror is available).
> MirrorBrain mailing lists also have a generic non-project-specific "networkers" list for talk between content providers and mirror admins, non-specific to any app.
The general mailing list is discuss@mirrorbrain.org (see http://mirrorbrain.org/communication/).
The MirrorBrain-specific mailing list is mirrorbrain@mirrorbrain.org.
Thanks again, and I'm looking forward to further discussion!
Peter
Re: Comments to your MirrorBrain notes
Date: 2010-02-22 05:32 pm (UTC)>... the best link almost always being either to Central Europe, to the U.S., or both.
I can think of cases where that is not true, or there are additional considerations. Some of the ISPs in South Africa have a peculiar bandwidth cap that applies only to traffic leaving the country, and otherwise it's unlimited.
Also raised in one of the other talks was cases with the Chinese firewall where entire datacentres had 10Gbit inside the country, but only 128Kbit leaving the country.
> In fact, routing data doesn't help with that. (Or I haven't found a way to make it useful.)
You need routing data from the perspective of the client and the data sources, not the MirrorBrain server.
Re: Comments to your MirrorBrain notes
Date: 2010-02-24 01:19 am (UTC)yes, which is why I said "almost always", and not "always" ;-) and which is why MirrorBrain prefers local mirrors.
What do you mean by "routing data from the perspective of the client"? Do you think you could you use it to detect the best mirror to send a client to? Or any other clever idea?
Thanks,
Peter