# Experiments with LVM-cache

Recently, we’ve frequently reached the I/O capacity of our RAID array during peak hours, meaning that increasingly often downloads were not limited by network speed, but by how fast our disks could deliver the data.

While we use a hardware RAID6 which does deliver pretty decent read speeds, we’re still using traditional magnetic hard drives, not SSDs. While these arrays easily deliver more than a Gigabyte per second when reading sequentially, this performance drops rapidly the more random the I/O gets, i.e. the more different files are requested at the same time. Of course, due to the nature of our FTP, which provides mirrors for a whole bunch of different projects, we do get a large amount of random I/O.

One solution to improve performance in this constellation is to use an additional cache on an SSD that caches the most frequently requested files or disk blocks. Most storage vendors implement something like that in their storage arrays nowadays, although it’s usually an optional and not exactly cheap feature – you usually pay a hefty license fee for that feature, and then you also have to buy prohibitively expensive SSDs from that vendor too to make any use of it. The better (and cheaper) alternative is to use a software solution to do the same thing. There are different implementations for SSD caching on Linux.

The SSD-caching implementation we chose to use was lvmcache. As the name suggests, lvmcache is integrated with the Linux Logical Volume Manager (LVM) that we use for managing the space on the big raid arrays anyways. The SSD is simply attached to a normal logical volume and then caches accesses to that volume by keeping track of which sectors are used most often and serving them from the cache-SSD.

Two basic modes are available: Write-back and Write-through. Write-Back writes blocks to the SSD-cache first and only syncs them to the disks some time later. While write-back has the advantage of increasing the write speed, it has the drawback that in this mode the SSD used for caching is vital – if it dies, the data on the disks would be left in an inconsistent and possible irrepairable state. To avoid data loss, some RAID level for the SSD would be required in this mode. However, we don’t really care about the write speed on the FTP – the only writes it sees are the updates of the mirrors, but those are few. More than 99% of all I/O is read requests from clients requesting some mirrored files. Because of that, we don’t really care about the write speed and instead use “write-through” mode. In this mode, all writes go to the underlying disk immediately, the SSD is only used for read caching. When the SSD dies, you lose the caching, but your data is still safe.

For testing, we borrowed a 1 TB Intel SSD. After one week of testing, we are impressed by the results. The following is a graph from our munin showing the utilization of the devices:

As you can see, we introduced the disk cache (nvme0n1) on the 7th. After about one day, the cache had filled up, and was now serving the magnitude of requests. As a result, the utilization on the disk arrays (sdb, sdc) dropped rapidly, from “pracitcally always 100%” to “30-50%” during peak hours.

If longterm performance is as good as these first test results suggest, we will permanently equip ftp.fau.de with a SSD for caching to allow faster downloads for you.

# New mirrors in the last months

Over the last 2.5 months we have added a few new mirrors. Amongst those are:

• CCC conference recordings by FeM. Here you can find recordings of almost all talks from the yearly Chaos Communication Congresses. This mirror also takes part in their official mirror network for these recordings under http://mirror.fem-net.de/CCC/
• CentOS. CentOS is a free rebuild of the Redhat Enterprise Linux packages (without the support of course).
• Eclipse, an IDE
• EPEL, Extra Packages for Enterprise Linux – EPEL packages a lot of additional software for Redhat Enterprise Linux or its clones
• the Qt project. It develops Qt, which is a great multiplatform application framework. There is a commercial company backing the project (currently Digia) that also offers commercial variants of the framework. Only the non-commercial releases are distributed over the mirror-network in which we’re taking part here. This is now one of the three mirrors that causes the most traffic on our FTP.
• The Document Foundation – TDF is responsible for the development of LibreOffice
• Trinity Desktop Environment. The TDE project is a fork of KDE3 and continues to develop a productive desktop environment that does not put looks above functionality.
• the VideoLAN project, which produces amongst other things a great free media player that plays almost everything (VLC media player). This is now another one of the three mirrors that causes the most traffic on our FTP.
• XBMC, a free media center software

# New FTP server

From this morning (October 01) on, ftp.uni-erlangen.de and ftp.fau.de point to new hardware.

The new machine is much more powerful than the old one, and we tried our best to make the mirroring setup as robust as possible. Hopefully, incosistent and outdated mirrors (because the mirror-job hung) are now a thing of the past. You can actually check this yourself: The FTP startpage lists the last successful run for all major mirrors, and there is also a page that shows how long the last mirror runs took and if they succeeded or not.

Another new feature on the startpage is that we show how much traffic the server has distributed the day before. We’re planning to further improve these stats, so that we can show the traffic per individual mirror. However, this is not finished yet.

The hardware specs of the new server are:

• HP DL360 G8 with Intel(R) Xeon(R) CPUs E5-2640 (2.50GHz, 2 sockets with 12 cores + 12 SMT threads)
• 64 GB RAM
• Two HP D2600 with 12x 2 TB NL-SAS hard discs each, on a SmartArray P822 RAID controller in RAID6
• 2x 10 GBit network interfaces (bonded / LACP)

There are some changes to the access methods:

• The FTP previously supported FTP with SSL. This no longer is the case.
• We now support HTTPS
• We now support RSYNC, though this is still experimental. We will disable RSYNC again if it proves to cause too much load on the server.
• We support IPv6 again. For technical reasons, the old FTP had no longer supported IPv6 connections for the last year.

In general, most of the mirrors that existed on the old FTP should be available. Though they now no longer reside in sub-sub-directorys like e.g. /pub/mirrors/ubuntu or /mirrors/ubuntu but simply in /ubuntu instead, symlinks have been set up from the old locations so that old links should still work. We did however clean out some mirrors that no longer made sense, for example:

• Cherokee (webserver) – cherokee no longer uses a mirror network, they now host their downloads on github (though they still link to outdated mirrors from their download page – really great work guys!)
• opensolaris – this project was killed off in 2009
• aminet – we haven’t been an official mirror since 2006

Should we accidently have dropped a mirror that you think still makes sense, or forgotten a symlink, feel free to contact us.

• centos
• epel
• FEMnet recordings from CCC conferences

Last but not least, the official name of the server is now “ftp.fau.de” instead of “ftp.uni-erlangen.de” – to fit with the “corporate identity” of FAU.

So go ahead and try the new FTP:

# Abkündigung privater Dateiaustauschbereiche

ftp.fau.de / ftp.uni-erlangen.de wird derzeit auf einem neuen, deutlich leistungsfähigeren System neu aufgesetzt. Damit wollen wir diesen beliebten Dienst auf eine solide Basis für die kommenden Jahre stellen.
Mit dem neuen System verschwindet aber auch ein Dienst. In der Vergangenheit gab es auf ftp.uni-erlangen.de neben den öffentlichen Mirrors für zahlreiche Linux-Distributionen und andere Open-Source-Software einige wenige “private Dateiaustauschbereiche”: Benutzer konnten sich mit Benutzernamen und Passwort einloggen und landeten dann in einem geschützten Bereich. Darüber war z.B. der Austausch von Dateien die zu gross fuer einen eMail-Anhang sind mit externen Projektpartnern möglich.
Dieser Dienst wird auf dem neuen System nicht mehr eingerichtet und somit zum 01.10.2013 ersatzlos eingestellt werden.