Thomas Zeiser

Some comments by Thomas Zeiser about HPC@RRZE and other things

Content

mpiexec + taskset: code snippet to generate config file

mpiexec + taskset: code snippet to generate config file

We usually use Pete Wyckoff’s mpiexec to start MPI processes from within batch jobs. The following code snippet may be used to pin individual MPI processes to their CPU. A similar snippet could be used to allow round-robin distribution of the MPI processes …

#!/bin/bash
# usage: run.sh 8 "0 2 4 6" ./a.out "-b -c"
#        start a.out with arguments -b -c
#        use 4 MPI processes
#        pin processes to CPUs 0,2,4,6

NUMPROC=$1
TASKCPUS="$2"
EXE=$3
ARGS=$4

TASKS=`echo $TASKCPUS | wc -w`
echo "running $NUMPROC MPI processes with $TASKS per node"

cat /dev/null > cfg.$PBS_JOBID
for node in `uniq $PBS_NODEFILE`; do
   for cpu in $TASKCPUS; do
      echo "$node : taskset -c $cpu $EXE $ARGS" >> cfg.$PBS_JOBID
      NUMPROC=$((NUMPROC - 1))
      if [ $NUMPROC -eq 0 ]; then
        break 2
      fi
   done
done
mpiexec -comm pmi -kill -config cfg.$PBS_JOBID