News zu ftp.fau.de (ftp.uni-erlangen.de)

Content

Evaluating Project Mirroring by Cost vs. Profit

People operate mirrors out of different reasons, this could be purely altruistic to support open source communities, or to reduce incoming traffic by providing local clients with an internal package source, or to tilt the in/out ratio of the internet uplink. What ever the reasons are, they are all associated with the usage of outbound traffic and local storage size, in addition to occupation of other hardware and administrative resources. We will consider traffic and storage requirements as the costs and outgoing traffic as the profit (to the general public or an organization) a mirror service provides.

Now looking around at what other mirror operators publish, it is very hard to find useful information to guide the mirror selection. Which means, that if you plan on hosting a new project, you have to rely on guesswork to estimate cost vs profit.

Although cost and profit are terms usually associated with currency based units, we will not estimate them using any such metric. We will consider Gigabytes as the unit for storage and traffic and therefore cost and profit.

By combining logged data from the ~50 mirrors we are currently hosting, we are in the position to quantify both cost and profit for each one of them. Be aware that the profit often depends on the geographic location, since many projects use a topology- and geography-aware load balancer like mirrorbrain.

Lets start of with the incoming traffic cost. The average number of GBs that need to be synced from the upstream mirror to ftp.fau.de per day:

Mirror Incoming Traffic
  GB per day (average)
kiwix 16.63
fedora 4.29
qtproject 3.73
debian 2.65
freebsd 2.63
ubuntu-ports 2.52
eclipse 1.83
cdn.media.ccc.de 1.80
deepin-cd 1.54
ubuntu 1.51
debian-cd 1.28
mageia 1.18
opensuse 1.01
macports 0.95
scientific 0.94
archlinux 0.90
trinity 0.87
macports/packages 0.83
netbsd 0.81
turnkeylinux 0.75
fosdem 0.67
gentoo 0.64
deepin 0.45
centos 0.32
xbmc 0.27
epel 0.27
packman 0.27
apache 0.21
openvz 0.20
osmc 0.19
mint 0.19
tdf 0.16
mint/iso 0.16
macports/distfiles 0.12
pclinuxos 0.09
openelec 0.07
opensource-dvd 0.07
ubuntu-releases 0.05
raspbmc 0.05
ripe.net 0.05
gnome 0.03
debian-backports 0.03
videolan 0.02
ctan 0.02
mint/packages 0.02
opencsw 0.02
CCC 0.02
knoppix-dvd 0.01
aminet 0.01
gimp 0.01
knoppix 0.00
grml 0.00
aptosid 0.00
mint/lmde-packages 0.00
gentoo-portage 0.00
macports/release 0.00
macports/trunk 0.00
putty 0.00
libreelec n/a

We calculated this by summing up all positive changes in mirror size, which is calculated and logged on a daily basis. Since we sync up to hourly, files that get synced and deleted within a day are not covered, nor are files that change.

How about storage costs? Since most mirrors stay around the same size, with a slight increase taking the average size works for most. But some have a tendency to constantly increase (or even decrease), this won’t be correct in all cases.

Mirror Storage Increment Total Storage Total Storage
  GB per day (average) GB (maximum) GB (average)
kiwix 4.19 5690 3977
cdn.media.ccc.de 1.75 3189 2338
scientific 0.82 1602 1328
fedora 0.88 1712 1304
ubuntu-ports 1.22 1214 1153
debian 0.44 1347 1078
CCC 0.02 1073 1073
mageia 0.65 1139 890
ubuntu 0.20 916 773
mint -0.37 873 652
eclipse 0.43 824 630
macports 0.11 727 593
freebsd -1.12 1224 579
qtproject 0.59 731 517
turnkeylinux 0.34 724 517
opensuse 0.28 552 441
mint/lmde-packages -0.44 671 440
macports/packages 0.19 576 399
fosdem 0.57 394 370
netbsd -0.13 467 368
gentoo 0.07 329 288
debian-cd -0.05 679 280
macports/distfiles -0.08 258 193
mint/iso 0.06 206 190
gnome 0.03 175 174
archlinux -0.08 532 160
trinity 0.09 304 160
epel 0.06 155 117
openvz -0.26 172 105
centos 0.08 141 104
pclinuxos -0.03 97 93
packman 0.03 99 82
ripe.net 0.04 100 76
apache 0.07 117 69
deepin 0.25 257 67
xbmc 0.01 73 52
opensource-dvd 0.07 79 46
aminet 0.00 44 43
tdf 0.01 49 38
deepin-cd -0.01 583 36
videolan 0.02 37 31
debian-backports -0.04 34 30
opencsw 0.01 31 28
ctan 0.00 33 27
ubuntu-releases -0.01 41 25
mint/packages 0.01 30 22
gimp 0.01 22 19
osmc 0.09 30 16
knoppix-dvd 0.00 16 16
openelec 0.00 23 12
grml 0.00 12 12
aptosid 0.00 8 9
libreelec n/a 6 6
knoppix 0.00 5 6
raspbmc 0.01 22 6
gentoo-portage 0.00 1 1
macports/release 0.00 0 0
macports/trunk 0.00 0 0
putty 0.00 0 0

Let’s have a look at the profit side of things, the outgoing traffic, and put that into relation with the costs. The scores on the right hands side, are calculated by dividing the average outgoing traffic by the average incoming traffic and by the average storage requirement:

Mirror Outgoing Traffic Incoming Traffic Storage Increment Total Storage Total Storage Score Based on Traffic Score Based on Storage
  GB per day (average) GB per day (average) GB per day (average) GB (maximum) GB (average) Outgoing vs Incoming Outgoing vs avg. Storage
osmc 478.40 0.19 0.09 30 16 2518 29.90
libreelec 56.90 n/a n/a 6 6 9.48
openelec 93.70 0.07 0.00 23 12 1339 7.81
videolan 176.60 0.02 0.02 37 31 8830 5.70
ctan 138.80 0.02 0.00 33 27 6940 5.14
ubuntu-releases 91.80 0.05 -0.01 41 25 1836 3.67
tdf 127.70 0.16 0.01 49 38 798 3.36
knoppix 13.40 0.00 0.00 5 6 2.23
mint/iso 350.20 0.16 0.06 206 190 2189 1.84
opensource-dvd 48.80 0.07 0.07 79 46 697 1.06
qtproject 494.50 3.73 0.59 731 517 133 0.96
grml 10.10 0.00 0.00 12 12 0.84
opensuse 365.00 1.01 0.28 552 441 361 0.83
centos 77.70 0.32 0.08 141 104 243 0.75
xbmc 35.60 0.27 0.01 73 52 132 0.68
pclinuxos 57.80 0.09 -0.03 97 93 642 0.62
deepin-cd 21.00 1.54 -0.01 583 36 14 0.58
knoppix-dvd 9.00 0.01 0.00 16 16 900 0.56
epel 57.10 0.27 0.06 155 117 211 0.49
packman 31.70 0.27 0.03 99 82 117 0.39
eclipse 238.90 1.83 0.43 824 630 131 0.38
mint/packages 6.90 0.02 0.01 30 22 345 0.31
archlinux 28.90 0.90 -0.08 532 160 32 0.18
macports/packages 66.40 0.83 0.19 576 399 80 0.17
fedora 195.00 4.29 0.88 1712 1304 45 0.15
kiwix 560.40 16.63 4.19 5690 3977 34 0.14
raspbmc 0.80 0.05 0.01 22 6 16 0.13
aptosid 1.00 0.00 0.00 8 9 0.11
aminet 4.70 0.01 0.00 44 43 470 0.11
macports/distfiles 18.90 0.12 -0.08 258 193 158 0.10
gentoo 26.70 0.64 0.07 329 288 42 0.09
turnkeylinux 46.10 0.75 0.34 724 517 61 0.09
cdn.media.ccc.de 185.50 1.80 1.75 3189 2338 103 0.08
opencsw 2.20 0.02 0.01 31 28 110 0.08
mageia 62.10 1.18 0.65 1139 890 53 0.07
apache 4.40 0.21 0.07 117 69 21 0.06
fosdem 22.70 0.67 0.57 394 370 34 0.06
deepin 3.60 0.45 0.25 257 67 8 0.05
ubuntu 41.00 1.51 0.20 916 773 27 0.05
debian 57.10 2.65 0.44 1347 1078 22 0.05
gimp 0.90 0.01 0.01 22 19 90 0.05
debian-cd 12.70 1.28 -0.05 679 280 10 0.05
CCC 25.80 0.02 0.02 1073 1073 1290 0.02
gnome 3.80 0.03 0.03 175 174 127 0.02
trinity 2.90 0.87 0.09 304 160 3 0.02
freebsd 8.20 2.63 -1.12 1224 579 3 0.01
openvz 1.00 0.20 -0.26 172 105 5 0.01
macports 4.80 0.95 0.11 727 593 5 0.01
scientific 9.30 0.94 0.82 1602 1328 10 0.01
ripe.net 0.40 0.05 0.04 100 76 8 0.01
netbsd 1.70 0.81 -0.13 467 368 2 0.00
ubuntu-ports 4.70 2.52 1.22 1214 1153 2 0.00
debian-backports 0.10 0.03 -0.04 34 30 3 0.00
mint 0.80 0.19 -0.37 873 652 4 0.00
mint/lmde-packages 0.30 0.00 -0.44 671 440 0.00
gentoo-portage 0.00 0.00 0.00 1 1 0.00
macports/release 0.00 0.00 0.00 0 0 n/a n/a
macports/trunk 0.00 0.00 0.00 0 0 n/a n/a
putty 0.00 0.00 0.00 0 0 n/a n/a

What can we learn from this? Small projects can produce a lot of traffic (osmc), especially if they tend to have a small mirror network (libreelec) or a humongous user base (videolan). Big mirrors on the other hand, often produce a lot of traffic in an absolute sense (kiwix, cdn.media.ccc.de), but they are unable to compensate for their storage requirement. For example, the CCC media mirror would have to constantly saturate 2 Gbit/s to get the same traffic/storage ratio as videolan. Although we love outgoing traffic, thankfully they do not. 🙂

Mirrors generating most traffic in 2015

Here are the most used mirrors in 2015:

Rank Mirror Traffic 2015 in TB Rank/Traffic 2014 (for comparison)
1 qtproject (Qt Toolkit) 199 1 / 172
2 kiwix 179 – / 0
3 mint/iso (Linux Mint ISOs) 177 10 / 31
4 opensuse 139 2 / 106
5 fedora 100 6 / 44
6 eclipse 97 4 / 86
7 cdn.media.ccc.de (Talk recordings from CCC conferences) 83 8 / 32
8 ctan (comprehensive TEX archive network) 63 7 / 40
9 tdf (The Document Foundation – LibreOffice) 48 5 / 48
10 videolan 40 3 / 88

Kiwix on rank 2 has only been mirrored since March 2015, so it generated that amount of traffic in only 9 and a half months – it might well take over the top spot next year. The Ubuntu release ISOs dropped out of the Top 10 for IPv4. As is to be expected, the CCC conference recordings generate extremely peaky traffic after CCC events – setting a new record high with 11.3 TB in just one day on December 30 (after 32C3). It might even have made a little bit more, but unfortunately the webserver ran out of threads because someone apparently distributed web-torrent-files with 64 KB of chunksize.

The list is slightly different for IPv6.

Rank Mirror IPv6 Traffic 2015 in TB Rank/Traffic 2014 (for comparison)
1 mint/iso (Linux Mint ISOs) 22.1 10 / 3.1
2 cdn.media.ccc.de (Talk recordings from CCC conferences) 15.2 6 / 4.7
3 kiwix 14.5 – / 0.0
4 fedora 11.5 7 / 3.6
5 qtproject (Qt Toolkit) 11.5 1 / 9.1
6 opensuse 10.8 3 / 6.6
7 debian 9.3 5 / 5.4
8 tdf (The Document Foundation – LibreOffice) 8.1 8 / 3.4
9 eclipse 7.4 4 / 5.8
10 ubuntu-releases (Ubuntu CD Images) 6.0 9 / 3.2

IPv6 traffic is up – from 7.86% over all mirrors in 2014 to 10.54% in 2015. Absolute IPv6 traffic is also up in 2015 for all mirrors in the Top 10. The share of IPv6 traffic varies greatly between the mirrors.

Changing filesystems: From XFS to EXT4

Since we moved the ftp to the (then) new hardware in October 2013, we had been using XFS as the filesystem on which we store all our mirror trees. All data resided in one single large XFS filesystem of 35 TB. That worked rather well until we updated the operating system on the machine in January.

After the update, the machine behaved extremely unstable – and in crass contrast to its 460 days uptime before the update. Sometimes all file I/O would stop for a few minutes (up to 30), and then suddenly continue as if nothing had happened. Those hangs happened in average twice a day. Sometimes the machine would also lock up completely and need to be reset.
During the hangs, the machine would log many of the following error-messages to the kernel log:

XFS: possible memory allocation deadlock in kmem_alloc (mode:0x8250)

The problems soon were traced to the new kernel 3.13 that came with the update. Simply booting the old 3.2 Kernel from before the update again would get rid of the weird hangs and lockups. We first tried to update the kernel to 3.16, but to no avail – it showed exactly the same problem.

Apparently, there was a bug in newer XFS versions. So I took the problem to the XFS mailing list. The responses were surprisingly quick and competent: Apparently there are situations where XFS will need large amounts of unfragmented kernel memory, and when such memory is not available, it will essentially block until there is. Which might be “when hell freezes over”. Until then there is no I/O to that filesystem anymore. The suggested workaround of increasing vm.min_free_kbytes and similar settings so the kernel would be more likely to have enough memory immediately available for XFS did not work out, hangs were still happening at an unacceptable rate – a FTP server that is unavailble for 2 times 10 minutes a day isn’t exactly my idea of a reliable service for the public. The XFS developers seemed to have some ideas on how to “do better”, but it would not be simple. I did not want to wait for them to implement something – and even after they did I would still have to run a handpatched kernel all the time, which I was trying to avoid. So a switch of filesystems was in order. That was probably a good idea, because according to a recent post on LKML the situation is still unchanged and no fix has been implemented yet.

We decided to switch to EXT4 in 64bit mode. Classic EXT4 would not be able to handle a filesystem of 35 TB, but in 64bit mode it can. That mode was implemented some time ago, but until recently the e2fsprogs-versions shipped with distributions were unable to create or handle filesystems with the 64bit option.

To switch filesystems with as little downtime as possible, we had to pull a few tricks.
Unfortunately, it is not possible to shrink XFS-filesystems, so we could not simply shrink the XFS filesystem to make room for an EXT4 filesystem. So first, we temporarily attached another storage-box with enough space for the new filesystem. We created a LVM-volume on it and an EXT4-filesystem in it. We then started to rsync all files over. That took about 3 days, but of course happened in the background without downtime. Next step was to temporarily stop all cronjobs that would update the mirrors, do another final rsync run, and then unmount the old filesystem and mount the new one. That naturally caused a few minutes of downtime, but still went without too many users noticing. We then killed off the old filesystem, and used LVM’s pvmove to move the data from the temporary storage-box back to the space that was previously occupied by the old filesystem. This again happened in the background and completed in about a day. We could then remove the temporary storage-box. So far, we had done the whole move with less than an hour of downtime. The only thing left to do was that we still had to resize the EXT4 to fill all the available space – the one created on the temporary storage-box had been smaller because it had a smaller capacity than our regular RAIDs.

This was where we hit the next snatch: Running resize2fs to do an online-resize of the filesystem would just send resize2fs into an endless loop. It turned out this is another known bug for large 64bit EXT4 filesystems: Apparently nobody had ever tested resizing 64bit EXT4 filesystems that actually used block numbers larger than 2^32, which is why both online- and offline-resize-functions would try to stick a 64 bit block number into 32 bit and then naturally explode. Lucky for us, it just went into an endless-loop instead of corrupting the filesystem by trimming some 64 bit block numbers to 32 bit…

As fixing the online-resize would again mean to compile and run a handpatched kernel, the only option really was to do an offline-resize with a recent (>= 1.42.12) e2fstools-version. Unfortunately, that meant another downtime of a little over an hour for resize and fsck. But in the end, it was successful.

After 8 days and numerous obstacles, we have successfully moved from XFS to EXT4 without data loss.

Mirrors generating most traffic in 2014

While we publish daily stats for all of our mirrors at https://ftp.fau.de/cgi-bin/show-ftp-stats.cgi, there is currently no aggregation over a whole year. So here are the most used mirrors over the whole year:

Rank Mirror Traffic 2014 in TB
1 qtproject (Qt Toolkit) 172
2 opensuse 106
3 videolan 88
4 eclipse 86
5 tdf (The Document Foundation – LibreOffice) 48
6 fedora 44
7 ctan (comprehensive TEX archive network) 40
8 cdn.media.ccc.de (Talk recordings from CCC conferences) 32
9 ubuntu-releases (Ubuntu CD Images) 32
10 mint/iso (Linux Mint ISOs) 31

The list is slightly different for IPv6.

Rank Mirror IPv6 Traffic 2014 in TB
1 qtproject (Qt Toolkit) 9.1
2 videolan 7.4
3 opensuse 6.6
4 eclipse 5.8
5 debian 5.4
6 cdn.media.ccc.de (Talk recordings from CCC conferences) 4.7
7 fedora 3.6
8 tdf (The Document Foundation – LibreOffice) 3.4
9 ubuntu-releases (Ubuntu CD Images) 3.2
10 mint/iso (Linux Mint ISOs) 3.1

Still only a small fraction of the traffic is via IPv6, 7.86% over all mirrors and the whole year. CTAN for some reason drops out of the IPv6 list (it would be at #17), its Top10 slot is taken over by Debian (#16 in the total list) which has an unusually large IPv6 share. That large share can be attributed mostly to the local Computer Science Department which has large pools with IPv6 capable Debian machines.

experimental geo-ip-statistics

Since the beginning of September, we’ve been creating experimental GeoIP-statistics to see where the users of ftp.fau.de come from. We’re drawing nice maps from the results. You can find these stats from our general ftp statistics page – for days where GeoIP-stats are available, you will find a link to them at the bottom of the page. Here is an example:

geoipstats-ftp.fau.de-20141229-worldmap

As you can see, unsurprisingly most of our users come from Germany. Besides the map there is also a table that show the respective percentages per country – in the map above, Germany accounts for about 50 percent of all accesses. Note that is counting the number of bytes transferred, not the number of accesses.

We also have a map from the stats per Bundesland within Germany:

geoipstats-ftp.fau.de-20141229-germany

However, this second map is really mostly guesswork for a number of technical reasons:

  • To do the actual dirty mapping work, we’re using GeoLite data created by MaxMind, available from http://www.maxmind.com. This is a free-to-use version of their commercial GeoIP-database, and since they need to make a living from their database, the free version is of course less accurate than the version available to their paying customers. It just returns “unknown” for the Bundesland in roughly about 30% of all cases.
  • Due to technical limitations on our side, we’re currently using a rather dusted version of the geoip-mapping library, that for example still has problems mapping IPv6 addresses (although it works sometimes). The quality of the mappings should improve significantly after we update to their current version in the first half of 2015.
  • Our logs are normally anonymized, i.e. we do not store the full IP address of a client, but trim (at least) the last octet of an IPv4 IP. This further reduces mapping quality.

Still, we do think it is a rather interesting experiment. While we were expecting most of our users to be from Germany, we found the actual percentage suprisingly high – we had expected some more usage from other european countries.

New mirrors in the last months

Over the last 2.5 months we have added a few new mirrors. Amongst those are:

  • CCC conference recordings by FeM. Here you can find recordings of almost all talks from the yearly Chaos Communication Congresses. This mirror also takes part in their official mirror network for these recordings under http://mirror.fem-net.de/CCC/
  • CentOS. CentOS is a free rebuild of the Redhat Enterprise Linux packages (without the support of course).
  • Eclipse, an IDE
  • EPEL, Extra Packages for Enterprise Linux – EPEL packages a lot of additional software for Redhat Enterprise Linux or its clones
  • the Qt project. It develops Qt, which is a great multiplatform application framework. There is a commercial company backing the project (currently Digia) that also offers commercial variants of the framework. Only the non-commercial releases are distributed over the mirror-network in which we’re taking part here. This is now one of the three mirrors that causes the most traffic on our FTP.
  • The Document Foundation – TDF is responsible for the development of LibreOffice
  • Trinity Desktop Environment. The TDE project is a fork of KDE3 and continues to develop a productive desktop environment that does not put looks above functionality.
  • the VideoLAN project, which produces amongst other things a great free media player that plays almost everything (VLC media player). This is now another one of the three mirrors that causes the most traffic on our FTP.
  • XBMC, a free media center software

More statistics

We have improved the publicly available statistics for our FTP server. You can now see how much traffic each individual mirror makes, and how much of the traffic was IPv6 vs. IPv4. The statistics are available at https://ftp.fau.de/cgi-bin/show-ftp-stats.cgi.

With this, you can see for example that the latest OpenSuSE 13.1 release had a quite significant impact on our FTP: The statistics for 19. November 2013 show that we distributed almost 9 TB for opensuse on that day, more than 5 times the traffic of all other mirrors combined.

New FTP server

From this morning (October 01) on, ftp.uni-erlangen.de and ftp.fau.de point to new hardware.

The new machine is much more powerful than the old one, and we tried our best to make the mirroring setup as robust as possible. Hopefully, incosistent and outdated mirrors (because the mirror-job hung) are now a thing of the past. You can actually check this yourself: The FTP startpage lists the last successful run for all major mirrors, and there is also a page that shows how long the last mirror runs took and if they succeeded or not.

Another new feature on the startpage is that we show how much traffic the server has distributed the day before. We’re planning to further improve these stats, so that we can show the traffic per individual mirror. However, this is not finished yet.

The hardware specs of the new server are:

  • HP DL360 G8 with Intel(R) Xeon(R) CPUs E5-2640 (2.50GHz, 2 sockets with 12 cores + 12 SMT threads)
  • 64 GB RAM
  • Two HP D2600 with 12x 2 TB NL-SAS hard discs each, on a SmartArray P822 RAID controller in RAID6
  • 2x 10 GBit network interfaces (bonded / LACP)

There are some changes to the access methods:

  • The FTP previously supported FTP with SSL. This no longer is the case.
  • We now support HTTPS
  • We now support RSYNC, though this is still experimental. We will disable RSYNC again if it proves to cause too much load on the server.
  • We support IPv6 again. For technical reasons, the old FTP had no longer supported IPv6 connections for the last year.

In general, most of the mirrors that existed on the old FTP should be available. Though they now no longer reside in sub-sub-directorys like e.g. /pub/mirrors/ubuntu or /mirrors/ubuntu but simply in /ubuntu instead, symlinks have been set up from the old locations so that old links should still work. We did however clean out some mirrors that no longer made sense, for example:

  • Cherokee (webserver) – cherokee no longer uses a mirror network, they now host their downloads on github (though they still link to outdated mirrors from their download page – really great work guys!)
  • opensolaris – this project was killed off in 2009
  • aminet – we haven’t been an official mirror since 2006

Should we accidently have dropped a mirror that you think still makes sense, or forgotten a symlink, feel free to contact us.

Some new mirrors were added:

  • centos
  • epel
  • FEMnet recordings from CCC conferences

Last but not least, the official name of the server is now “ftp.fau.de” instead of “ftp.uni-erlangen.de” – to fit with the “corporate identity” of FAU.

So go ahead and try the new FTP:

Abkündigung privater Dateiaustauschbereiche

ftp.fau.de / ftp.uni-erlangen.de wird derzeit auf einem neuen, deutlich leistungsfähigeren System neu aufgesetzt. Damit wollen wir diesen beliebten Dienst auf eine solide Basis für die kommenden Jahre stellen.
Mit dem neuen System verschwindet aber auch ein Dienst. In der Vergangenheit gab es auf ftp.uni-erlangen.de neben den öffentlichen Mirrors für zahlreiche Linux-Distributionen und andere Open-Source-Software einige wenige “private Dateiaustauschbereiche”: Benutzer konnten sich mit Benutzernamen und Passwort einloggen und landeten dann in einem geschützten Bereich. Darüber war z.B. der Austausch von Dateien die zu gross fuer einen eMail-Anhang sind mit externen Projektpartnern möglich.
Dieser Dienst wird auf dem neuen System nicht mehr eingerichtet und somit zum 01.10.2013 ersatzlos eingestellt werden.