Content

Unplanned outage on June 03

On June 03, 2020, at around 16:15 local time (14:15 UTC), disk accesses on ftp.fau.de became extremely slow. Less than an hour later, any attempts to access the disks with the ftp-data on it failed. Investigation revealed that the big RAID controller that manages all the external disk enclosures for the data had stopped responding completely.
While the failed controller could temporarily be brought back by a powercycle at around 18:00 local time (16:00 UTC), it failed again within 10 minutes of booting the machine.
Unfortunately, there was no compatible replacement for the failed controller onsite. A replacement has been ordered and shipped, but has not arrived yet. As all parcel services are severely overloaded due to the Corona crisis, it is currently unclear when it will arrive.

We were able to bring ftp.fau.de partially back after noon on June 04: It seems the broken controller does not crash as long as it does not get too much load. We have therefore had to disable automatic updates of all mirrors for now. They will mostly remain at the version they had at around 2020-06-03 14:30 UTC. We have however updated a few select mirrors manually.

The controller also is significantly slower than normal, even if it has a significantly lower workload than usual. This is mostly because we have disabled any write caching on it, which indirectly automatically slows the throughput it can achieve to a crawl. While many accesses can be handled by the big SSD that serves as a cache (and is working perfectly fine), in those cases where a fetch from the spinning hard discs is needed because data is not in the cache, these will be significantly slower than usual.

We are sorry for the inconvenience and trying our best to return to full service ASAP.

We will update this article as needed.

Update 1 @2020-06-06 08:30: Parcel tracking now says that our delivery has arrived in our city and will be delivered to us on Monday, so we expect to be back in business by Monday evening.

Update 2 @2020-06-07 08:00: While we are not back to our usual sync schedules yet, all mirrors should be updated at least once a day.

Update 3 @2020-06-08 14:00: The replacement controller has arrived.

Update 4 @2020-06-08 22:00: The replacement-controller is working fine. All mirrors are current again, and normal update intervals have been resumed.
While we were working on the machine anyways, we also upgraded the main memory from 64 to 128 GB.

Mirrors generating most traffic in 2019

The total traffic of ftp.fau.de increased from 3.22 PB in 2018 to 3.52 PB in 2019, an unusually small increase of only 9%.

Rank Mirror Traffic 2019 in TB Rank/Traffic 2018 (for comparison)
1 qtproject (Qt Toolkit) 511 2 / 445
2 kiwix (offline Wikipedia) 458 1 / 476
3 mint/iso 260 5 / 255
4 lineageos (Free and Open-Source Android distribution) 246 3 / 327
5 eclipse 162 4 / 266
6 opensuse 138 7 / 119
7 fedora 136 6 / 137
8 cdn.media.ccc.de (Talk recordings from CCC and related conferences) 120 – / 94
9 centos 117 9 / 109
10 osmc (Open Source Media Center) 112 8 / 113

The percentage of IPv6 traffic increased a tiny bit again, to 25% (0.87 PB), after 22% in 2018, 22% in 2017, 14% in 2016 and 11% in 2015.

There is little change in the Top 10 overall, with traffic volumes staying pretty much the same, only some mirrors swapped their places in the table.

Our mirror of cdn.media.ccc.de managed to reach the Traffic Top 10 again, but it was a close call. This mirror makes most of its yearly traffic at the end of December / beginning of January, when the recordings of the annual Chaos Communication Congress are put online. On the other hand, the CTAN mirror that was on rank 10 last year did not make the Top 10 this year, missing it by a few TB (106).

A notable new entry that would be on rank 11 is F-Droid, a community-maintained Android software repository only hosting free/libre software (sort of an alternative to Googles “Play Store”). We started mirroring this at the end of 2018, but only recently they added functionality that makes the clients use the mirrors more often. As a result, this mirror has seen some a lot more usage in recent months, and is likely to reach the Top 10 in 2020.

Mirrors generating most traffic in 2018

The total traffic of ftp.fau.de increased from 2.51 PB in 2017 to 3.22 PB in 2018, a 28% increase.

Rank Mirror Traffic 2018 in TB Rank/Traffic 2017 (for comparison)
1 kiwix (offline Wikipedia) 476 1 / 371
2 qtproject (Qt Toolkit) 445 2 / 274
3 lineageos (Free and Open-Source Android distribution) 327 3 / 203
4 eclipse 266 7 / 147
5 mint/iso 255 5 / 190
6 fedora 137 8 / 103
7 opensuse 119 5 / 152
8 osmc (Open Source Media Center) 113 6 / 149
9 centos 109 11 / 69
10 ctan (comprehensive TEX archive network) 95 10 / 93

Even though the absolute amount of IPv6 traffic increased a bit, its percentage of all traffic stagnated, with 22% (0.72 PB) in 2018, after 22% in 2017, 14% in 2016 and 11% in 2015.

Our mirror of cdn.media.ccc.de is no longer in the Traffic Top 10, it only ranked in 11th place with a measly 94 TB, a large part of that at the end of December / beginning of January, when the recordings of the annual Chaos Communication Congress are put online.

Mirrors generating most traffic in 2017

The total traffic of ftp.fau.de increased from 1.89 PB in 2016 to 2.51 PB in 2017, by 33%. 203 TB alone was accounted to one of our newest additions from late may 2017: lineageos, a free and open-source android distribution and successor of cyanogenmod (which was discontinued).

Rank Mirror Traffic 2017 in TB Rank/Traffic 2016 (for comparison)
1 kiwix (offline Wikipedia) 371 2 / 262
2 qtproject (Qt Toolkit) 274 1 / 264
3 lineageos (Free and Open-Source Android distribution) 203 (since late may) – / –
4 mint/iso 190 5 / 134
5 opensuse 152 4 / 145
6 osmc (Open Source Media Center) 149 3 / 172
7 eclipse 147 6 / 131
8 fedora 103 7 / 115
9 cdn.media.ccc.de (Talk recordings from CCC conferences) 99 8 / 91
10 ctan (comprehensive TEX archive network) 93 9 / 76

The total IPv6 traffic increased significantly, to 22% (0.55 PB) of all traffic in 2017, from 14% in 2016 and 11% in 2015.