News zu ftp.fau.de (ftp.uni-erlangen.de)

Content

Mirrors generating most traffic in 2014

While we publish daily stats for all of our mirrors at https://ftp.fau.de/cgi-bin/show-ftp-stats.cgi, there is currently no aggregation over a whole year. So here are the most used mirrors over the whole year:

Rank Mirror Traffic 2014 in TB
1 qtproject (Qt Toolkit) 172
2 opensuse 106
3 videolan 88
4 eclipse 86
5 tdf (The Document Foundation – LibreOffice) 48
6 fedora 44
7 ctan (comprehensive TEX archive network) 40
8 cdn.media.ccc.de (Talk recordings from CCC conferences) 32
9 ubuntu-releases (Ubuntu CD Images) 32
10 mint/iso (Linux Mint ISOs) 31

The list is slightly different for IPv6.

Rank Mirror IPv6 Traffic 2014 in TB
1 qtproject (Qt Toolkit) 9.1
2 videolan 7.4
3 opensuse 6.6
4 eclipse 5.8
5 debian 5.4
6 cdn.media.ccc.de (Talk recordings from CCC conferences) 4.7
7 fedora 3.6
8 tdf (The Document Foundation – LibreOffice) 3.4
9 ubuntu-releases (Ubuntu CD Images) 3.2
10 mint/iso (Linux Mint ISOs) 3.1

Still only a small fraction of the traffic is via IPv6, 7.86% over all mirrors and the whole year. CTAN for some reason drops out of the IPv6 list (it would be at #17), its Top10 slot is taken over by Debian (#16 in the total list) which has an unusually large IPv6 share. That large share can be attributed mostly to the local Computer Science Department which has large pools with IPv6 capable Debian machines.

experimental geo-ip-statistics

Since the beginning of September, we’ve been creating experimental GeoIP-statistics to see where the users of ftp.fau.de come from. We’re drawing nice maps from the results. You can find these stats from our general ftp statistics page – for days where GeoIP-stats are available, you will find a link to them at the bottom of the page. Here is an example:

geoipstats-ftp.fau.de-20141229-worldmap

As you can see, unsurprisingly most of our users come from Germany. Besides the map there is also a table that show the respective percentages per country – in the map above, Germany accounts for about 50 percent of all accesses. Note that is counting the number of bytes transferred, not the number of accesses.

We also have a map from the stats per Bundesland within Germany:

geoipstats-ftp.fau.de-20141229-germany

However, this second map is really mostly guesswork for a number of technical reasons:

  • To do the actual dirty mapping work, we’re using GeoLite data created by MaxMind, available from http://www.maxmind.com. This is a free-to-use version of their commercial GeoIP-database, and since they need to make a living from their database, the free version is of course less accurate than the version available to their paying customers. It just returns “unknown” for the Bundesland in roughly about 30% of all cases.
  • Due to technical limitations on our side, we’re currently using a rather dusted version of the geoip-mapping library, that for example still has problems mapping IPv6 addresses (although it works sometimes). The quality of the mappings should improve significantly after we update to their current version in the first half of 2015.
  • Our logs are normally anonymized, i.e. we do not store the full IP address of a client, but trim (at least) the last octet of an IPv4 IP. This further reduces mapping quality.

Still, we do think it is a rather interesting experiment. While we were expecting most of our users to be from Germany, we found the actual percentage suprisingly high – we had expected some more usage from other european countries.

New mirrors in the last months

Over the last 2.5 months we have added a few new mirrors. Amongst those are:

  • CCC conference recordings by FeM. Here you can find recordings of almost all talks from the yearly Chaos Communication Congresses. This mirror also takes part in their official mirror network for these recordings under http://mirror.fem-net.de/CCC/
  • CentOS. CentOS is a free rebuild of the Redhat Enterprise Linux packages (without the support of course).
  • Eclipse, an IDE
  • EPEL, Extra Packages for Enterprise Linux – EPEL packages a lot of additional software for Redhat Enterprise Linux or its clones
  • the Qt project. It develops Qt, which is a great multiplatform application framework. There is a commercial company backing the project (currently Digia) that also offers commercial variants of the framework. Only the non-commercial releases are distributed over the mirror-network in which we’re taking part here. This is now one of the three mirrors that causes the most traffic on our FTP.
  • The Document Foundation – TDF is responsible for the development of LibreOffice
  • Trinity Desktop Environment. The TDE project is a fork of KDE3 and continues to develop a productive desktop environment that does not put looks above functionality.
  • the VideoLAN project, which produces amongst other things a great free media player that plays almost everything (VLC media player). This is now another one of the three mirrors that causes the most traffic on our FTP.
  • XBMC, a free media center software

More statistics

We have improved the publicly available statistics for our FTP server. You can now see how much traffic each individual mirror makes, and how much of the traffic was IPv6 vs. IPv4. The statistics are available at https://ftp.fau.de/cgi-bin/show-ftp-stats.cgi.

With this, you can see for example that the latest OpenSuSE 13.1 release had a quite significant impact on our FTP: The statistics for 19. November 2013 show that we distributed almost 9 TB for opensuse on that day, more than 5 times the traffic of all other mirrors combined.

New FTP server

From this morning (October 01) on, ftp.uni-erlangen.de and ftp.fau.de point to new hardware.

The new machine is much more powerful than the old one, and we tried our best to make the mirroring setup as robust as possible. Hopefully, incosistent and outdated mirrors (because the mirror-job hung) are now a thing of the past. You can actually check this yourself: The FTP startpage lists the last successful run for all major mirrors, and there is also a page that shows how long the last mirror runs took and if they succeeded or not.

Another new feature on the startpage is that we show how much traffic the server has distributed the day before. We’re planning to further improve these stats, so that we can show the traffic per individual mirror. However, this is not finished yet.

The hardware specs of the new server are:

  • HP DL360 G8 with Intel(R) Xeon(R) CPUs E5-2640 (2.50GHz, 2 sockets with 12 cores + 12 SMT threads)
  • 64 GB RAM
  • Two HP D2600 with 12x 2 TB NL-SAS hard discs each, on a SmartArray P822 RAID controller in RAID6
  • 2x 10 GBit network interfaces (bonded / LACP)

There are some changes to the access methods:

  • The FTP previously supported FTP with SSL. This no longer is the case.
  • We now support HTTPS
  • We now support RSYNC, though this is still experimental. We will disable RSYNC again if it proves to cause too much load on the server.
  • We support IPv6 again. For technical reasons, the old FTP had no longer supported IPv6 connections for the last year.

In general, most of the mirrors that existed on the old FTP should be available. Though they now no longer reside in sub-sub-directorys like e.g. /pub/mirrors/ubuntu or /mirrors/ubuntu but simply in /ubuntu instead, symlinks have been set up from the old locations so that old links should still work. We did however clean out some mirrors that no longer made sense, for example:

  • Cherokee (webserver) – cherokee no longer uses a mirror network, they now host their downloads on github (though they still link to outdated mirrors from their download page – really great work guys!)
  • opensolaris – this project was killed off in 2009
  • aminet – we haven’t been an official mirror since 2006

Should we accidently have dropped a mirror that you think still makes sense, or forgotten a symlink, feel free to contact us.

Some new mirrors were added:

  • centos
  • epel
  • FEMnet recordings from CCC conferences

Last but not least, the official name of the server is now “ftp.fau.de” instead of “ftp.uni-erlangen.de” – to fit with the “corporate identity” of FAU.

So go ahead and try the new FTP:

Abkündigung privater Dateiaustauschbereiche

ftp.fau.de / ftp.uni-erlangen.de wird derzeit auf einem neuen, deutlich leistungsfähigeren System neu aufgesetzt. Damit wollen wir diesen beliebten Dienst auf eine solide Basis für die kommenden Jahre stellen.
Mit dem neuen System verschwindet aber auch ein Dienst. In der Vergangenheit gab es auf ftp.uni-erlangen.de neben den öffentlichen Mirrors für zahlreiche Linux-Distributionen und andere Open-Source-Software einige wenige “private Dateiaustauschbereiche”: Benutzer konnten sich mit Benutzernamen und Passwort einloggen und landeten dann in einem geschützten Bereich. Darüber war z.B. der Austausch von Dateien die zu gross fuer einen eMail-Anhang sind mit externen Projektpartnern möglich.
Dieser Dienst wird auf dem neuen System nicht mehr eingerichtet und somit zum 01.10.2013 ersatzlos eingestellt werden.