Thomas Zeiser

Some comments by Thomas Zeiser about HPC@RRZE and other things

Content

Tag 1 der Woody-Installation- Auf in neue Welten oder der Kampf mit dem Aufzug

… nach langem Warten ist es endlich so weit: 8.6 TFlop/s Rechenleistung rollen an: das neue HPC-Cluster von Bechtle/HP mit Intel Woodcrest CPUs für die HPC-Crew an der Uni-Erlangen. Berichte über die Vertragsunterzeichnung gab es ja schon in den Erlanger Nachrichten sowie in den VDI-Nachrichten. Ein kurzer technischer Überblick findet sich in der aktuellen BI (BenutzerInformation).

Doch bis in den Rechnerraum ist es ein weiter Weg (1. OG) mit so einigen Hürden: Aufzug der im 12. OG fest steckt und Aufzugtechniker, die Stunden benötigen, bis sie an die FAU kommen. In die beiden kleinen Aufzüge passen die Racks wegen einigen Zentimetern nicht hinein. Tierisch nervende Bedienstete der Informatik, die nicht verstehen können oder wollen, dass bei so einer Anlieferung und einem nicht funktionierenden Aufzug einige sperrige Teile etwas im “Weg” stehen (müssen).

Im Rechnerraum daher noch gähnende Leere …

ah, unser neuer Rechner? Nein, nicht wirklich, aber vielleicht die Nutzer von übermorgen! Schlulklasse@RRZE

Und auch mit Aufzug und durch das Mittagessen gestärkt ist es mühsam

neuer Tag? Noch nicht, aber Stromsparmassnahmen …

So langsam tut sich nun wirklich etwas

wenngleich ein Rack mit rund 800 kg nicht so leicht dirigierbar ist

Probleme? Seit dem Aufzug eigentlich nicht mehr 🙂 Und alle Unsicherheiten über die Tragfähigkeit der Bodenplatten lassen sich schnell ausräumen.

… und relativ schnell ist die Hardware dann auch weitgehend an der Stelle, an die sie soll.

Damit ist aber natürlich die Arbeit noch lange nicht getan. Racks sauber ausrichten, verschauben, und noch ein “wenig” Konfigurationsarbeit … und der nächste Tag kommt bestimmt.

Bis zum regulären Benutzerbetrieb wird aber wohl noch einige Zeit vergehen 🙂

Abschluss von Aufbau und Konfiguration, Leistungsüberprüfung, Stabilitätstest, RRZE-Finetuning, etc. steht ja alles noch an … und nicht zu vergessen die Dokumentation auf den RRZE-Webseiten, damit nicht jeder Benutzer immer gleich bei uns anruft

ICMMES-2006

The 3rd International Conference for Mesoscopic Methods in Engineering and Science (http://www.icmmes.org) is currently taking place in Hampton/VA, USA …

People from Erlangen are very active this year:

  • a tutorial given by Gerhard Wellein on contemporary cache-based CPUs and basic optimization strategies for simple LBM kernels
  • an invited talk by Gerhard Wellein et al. about bleading edge optimizations (e.g. “cache oblivious blocking”)
  • and further contributions from Klaus Iglberger (Simulation of moving particles in 3D), Nils Thürey (“Adaptive simulation of open water free surfaces”), and Thomas Zeiser (Microscale flow analysis of trabecular vertebral bone)
  • and a joint poster of University of Amsterdam, NEC CCRLE, HLRS, CAB and RRZE.

MpCCI 3.0.5 released — but not yet working at RRZE

A new version of MpCCI (3.0.5) has been released. See www.mpcci.org for more details. Many things changed — in particular, the MpCCI library itself should be much more encapsulated so that it should be completely independent of the MPI version used within the application program.

However, owing to licensing problems, the new version could not yet be tested on the HPC systems of RRZE … Well, the solution was rather trivial: the license file did not contain the FQN of the server and therefore checking out or even testing the license from a remote host did not work if the FQN was used there, i.e. the solution was just to replace the shortend hostname by the full hostname (including domain) or also possible by the IP address. After restarting the license server, the licensensing problems disappeard

In the meantime (early August 2006), MpCCI 3.0.5 is also available on the LRZ machines …

However, there seem to be still some problems on the application side — probably not a big surprise as the internal structure of MpCCI 3.0.5 and the API have completely changed …

… especially on SGI Altix systems (or other machines where CPUsets are used), a start mechanism based on ssh cannot be used as CPUsets would be escaped and “batch” processes start running on interactive/boot CPUsets …

Using NEC SX-8 at HLRS

The NEC SX-8 is a nice vector machine. The compilers produce a lot of compile time information concerning optimization and vectorization.

Some unsorted notes to make life easier…

  • NEC SX series uses big endian, the “fronend system”, an Itanium2 system (NEC TX-7) however little endian. Fortran programmers have several options for automatic conversion of unformated data:
    • on the IA64 side (for executables generated with the Intel compiler): setenv F_UFMTENDIAN=big makes your program read/write big endian data
    • on the NEC SX side: setenv F_UFMTENDIAN u[,u...] to read/write little endian data; u[,u,...] are the units to convert.
  • by default, the NEC SX-8 Fortran runtime library does not read/write unformated records which are larger than 2 GB. setenv F_EXPRCW u,[u,...] enables processing of file records that are over 2GB in length. When this environment variable is specified, both record control words and End of Record markers are expanded to 8-byte quantities. F_EXPRCW and F_UFMTENDIAN cannot be used for the same unit.

MpCCI auf SGI Altix (3)

MpCCI auf SGI Altix (3)

Tests to use the client-server mode with SGI MPT … — a never ending tragedy:

  • link your application with: ccilink -client -nompilink ..... -lmpi
  • produce the procgroup file with ccirun -server -norun ex2.cci
  • run your simulation: 
    /opt/MpCCI/mpiexec/bin/mpiexec -server &
/opt/MpCCI/mpiexec/bin/mpiexec -config=ccirun_server.procgroup >& s & x1 & x2 < /dev/null
sleep 10

    • The number of CPUs requested must be at least as large as the number of server processes, i.e. those started with mpiexec.

    If you use mpich instead of MPT all processes have to be started with mpiexec. As a consequence, the number of requested CPUs must be equal to the total number of processes. The PBS-TM interface used by mpiexec does not allow to overbook CPUs.

    … after many hours of testing: IT SEEMS THAT MpCCI WITH SGI-MPT DOES *NOT* WORK RELIABLY AT ALL … mpi_comm_rank==0 on all processes 🙂 despite using the correct mpif.h files and mpiruns for the server and client applications.

    My current conclusions:

    • MpCCI does not support SGI MPT natively
    • using mpich on SGI Altix for all communications is NO option as benchmarks showed that CFD applications are slower by a factor of 2 or more when using mpich instead of MPT
    • using MpCCI in client-server mode also seems not to work (see above)

    That means, MpCCI is not at all usable on SGI Altix. Sorry for those who rely on it.

What are potential applications to exploit the GlobusToolkit / Unicore installation at RRZE?

What are potential applications to exploit the GlobusToolkit / Unicore installation at RRZE?

Although there are still some difficulties in getting the GlobusToolkit installation running, I’m wodering who will use it.

The GlobusToolkit is basically just a framework. Therefore, users probably have to spend some effort to make their applications GT4-aware … Just using GSIftp or SGIssh probably is not a big progress …

With Unicore, the situation might be slightly better as there is a nice GUI. However, also this GUI has to be extended by plugins written in Java to realy make progress. Just using the shell plugin does not really help. But at least, the Unicore client can easily be installed on the user’s system without requiering as host certificate or installing many hunderds MB as it is the case for GT4.

But anyway, my (current) feeling is that all grid services will have the same future as advanced debugger: ssh/slogin/scp will remain the tool of choise as printf will remain the most popular "debugger". But I’d be gladed to be convinced otherwise (by practial examples).

SGI Altix extension

Recently, the SGI Altix at RRZE has been extended. We now have a batch-only system altix-batch (an SGI Altix 3700 with 32 CPUs and 128 GB shared memory) and a front-end system altix (an SGI Altix 330 with 16 CPUs and 32 GB shared memory; 4 CPUs + 8 GB are used as login partition (boot cpuset) – the remaining ones are also used for batch processing).

An important thing to note is that the new machine has only half the amount of memory per CPU as the "old" one. As the cpusets introduced with SuSE SLES9/SGI ProPack4.x do not have all the features known from the old SGI Origin cpusets, in particular policy kill is missing, the systems starts swapping as soon as one process exceeds the amount of memory available in its cpuset. As a results, the complete system becomes un-responsive.

Therefore, it is very important to request the correct machine or to specify the amount of memory required in addition to the number of CPUs. Also the amount of interactive work is now much more limitated as we now have a login partition (boot cpuset) which only have access to 4 CPUs and 8 GB of memory!

Check the official web page of the SGI Altic Systems at RRZE for more details and the correct syntax for specifying recource requirements.

Handling BibTeX files

BibTeX is the bibliography format if you are working with LaTeX. Emacs has a very nice mode for editing BibTeX files, in particular if you add

(setq bibtex-maintain-sorted-entries 1)
(setq bibtex-clean-entry-zap-empty-opts nil)

to your .emacs file. However, sometimes it can be easier to have a specialized GUI. People suggested

Both tools are said to be easy to use and to have many options for importing (e.g. directly from electronic data bases such as INSPEC) or exporting (e.g. to endnote). However, I never had time to test them …

STAR-CD 4.0 available

Since resently, STAR-CD 4.0 is available for x86 and x86_64. Unfortunately, the currently builds again require the ABSOFT compiler to include user subroutines 🙂 Further builds and support or IA64 and MS Windows hopefully apprear in the following months …

STAR-CD 4.0 has already installed on Cluster32. All STAR-CD users are encouraged to test this new version. Iy you are using user subroutines, some changes are probably required as CD-adapco moved from Fortran77 to Fortran90 and eliminated one common data structure. Check the documentation for more details.

For those, running STAR-CD on their local systems, the installation files are available at the usual place.