The batch scheduler on the Cray XC is PBSpro. The basic functionality is similar to LoadLeveler:

• The user includes a number of directives at the start of the job script which provide information about which queue the job should run in, the location of the output and error files, the resources the jobs needs, etc.
• A number of command line tools are provided for submitting, viewing and managing the job.

These are the topics covered:

Main command line tools

The table summarises the main PBSpro user commands and their LoadLeveler equivalents.

Access to the job output and error files while the job is running

So that users are able to access the job output and error files while the job is running, ECMWF has provided the qcat command.

usage: qcat [-h] [-o | -e] [-f] JID
Access the job output and error files while the job is running
positional arguments:
  JID           The job ID
optional arguments:
  -h, --help    show this help message and exit
  -o, --ouptut  Get the stdout of the job. This is the default
  -e, --error   Get the stderr
  -f, --follow  Get the live output as it goes

Batch queues on the ECMWF Cray XC systems

As on the IBM systems, a number of batch queues have been defined. The basic queue names remain the same as on the IBM. The specifications of the main queues are given in the table.

Migrating LoadLeveler jobs to PBSpro

To help with the migration of batch jobs from LoadLeveler to PBSpro, ECMWF has provided the ll2pbs command:

usage: ll2pbs [-h] [-q] [-s] [-i INSCRIPT] [-o OUTSCRIPT] [-f]
Job translator from LoadLeveler to PBSPro
optional arguments:
  -h, --help            show this help message and exit
  -q, --quiet           Do not produce warning or error messages on stderr
  -s, --strict          Strict Translation. Fail if a directive is not
                        supported. Default is to skip the unsupported
                        directive
  -i INSCRIPT, --inscript INSCRIPT
                        Input script. By default reads stdin
  -o OUTSCRIPT, --outscript OUTSCRIPT
                        Output translated script. By default writes to stdout
  -f, --force           Overwrite the output file if it exists

This command can be used for the initial translation of simple scripts from Loadleveler to PBSpro.  It returns warnings for any directives that are not recognised or cannot be converted:

% ll2pbs -i job_ll.cmd -o job_pbs.cmd
WARNING: directive comment not supported, skipping...
WARNING: directive cpu_limit not supported, skipping...
WARNING: No variables allowed in the output file definition in PBS. Reverting to default values...
WARNING: No variables allowed in the error file definition in PBS. Reverting to default values...

The LoadLeveler file used as the input to this command and the PBSpro output file produced are:

#!/bin/ksh
#@ shell            = /usr/bin/ksh
#@ class            = ns
#@ job_type         = serial
#@ job_name         = pi
#@ comment          = User Support example pi (serial)
#@ output           = $(job_name).$(schedd_host).$(jobid).out
#@ error            = $(job_name).$(schedd_host).$(jobid).out
#@ notification     = error
#@ resources        = ConsumableCpus(1) ConsumableMemory(100mb)

#@ cpu_limit        = 00:16:00
#@ initialdir       = /scratch/ms/co/uid
#@ wall_clock_limit = 00:18:00
#@ queue

#!/bin/ksh
#PBS -S /usr/bin/ksh
#PBS -q ns

#PBS -N pi



#PBS -m a
#PBS -l EC_memory_per_task=100mb
#PBS -l EC_threads_per_task=1


#PBS -l walltime=00:18:00


cd /scratch/ms/co/uid

PBSpro also provides the nqs2pbs command.  This  utility converts an existing NQS job script to work with PBSpro and NQS.  The existing script is copied and PBS directives inserted prior to each NQS directive in the original script.  See 'man nqs2pbs' for more details.

PBSpro job header keywords

job_cpu_limit=<hh:mm:ss>

ECMWF custom PBSpro directives

In addition to the standard PBSpro directives, ECMWF has defined a number of custom directives to help the user define the geometry of the job. A full list of all ECMWF custom directives can be found at ECMWF PBSpro. Some of the more commonly used options together with their LoadLeveler equivalents are listed in the table:

Associated with each of the ECMWF custom directives is an environment variable of the same name.  These environment variable can be used in job scripts and, in particular, to specify the options to the aprun command.

PBSpro Job examples

Serial job example

• Serial jobs should be submitted to the ns queue.
• There is no need to specify any further job geometry requirements.
• For jobs requiring more than the default 1.25 GBytes of memory per task, the memory requirement should be set with the EC_memory_per_task directive.
  
  

#PBS -N Hello_serial
#PBS -q ns

./Hello_serial

Pure OpenMP job example

• Pure OpenMP jobs can be submitted to either:
  • queue np if EC_threads_per_task > 18 x EC_hyperthreads
  • queue nf if EC_threads_per_task <=  18 x EC_hyperthreads
• Set EC_total_tasks=1 to specify that this does not use MPI
• Set EC_threads_per_task to the number of OpenMP threads required.  This will be the value needed for the OMP_NUM_THREADS environment variable.
• Choose whether or not to use hyperthreading by setting EC_hyperthreads.
  
  

If the job is submitted to the np queue then the executable must be run with the aprun command.

#PBS -N HelloOpenMP
#PBS -q np
#PBS -l EC_total_tasks=1
#PBS -l EC_threads_per_task=72
#PBS -l EC_hyperthreads=2

export OMP_NUM_THREADS=$EC_threads_per_task
aprun -N $EC_tasks_per_node -n $EC_total_tasks -d $OMP_NUM_THREADS -j $EC_hyperthreads ./HelloOpenMP

For a pure OpenMP fractional jobs running in the nf queue, the executable can be called directly:

#PBS -N HelloOpenMP
#PBS -q nf
#PBS -l EC_total_tasks=1
#PBS -l EC_threads_per_task=8
#PBS -l EC_hyperthreads=2

export OMP_NUM_THREADS=$EC_threads_per_task
./HelloOpenMP

Pure MPI job example

• Pure MPI jobs can be submitted to either:
  • queue np if EC_total_tasks > 18 x EC_hyperthreads
  • queue nf if EC_total_tasks <= 18 x EC_hyperthreads
• Set EC_total_tasks to the total number of MPI tasks to be used.
• Optionally set EC_threads_per_task=1 to specify that no OpenMP threads will be used.
• Optionally choose the amount of memory per MPI task with the EC_memory_per_task directive.
• Choose whether or not to use hyperthreading by setting EC_hyperthreads.
• The executable must be run with the aprun command
  
  

#PBS -N HelloMPI
#PBS -q np
#PBS -l EC_total_tasks=72
#PBS -l EC_hyperthreads=2

aprun -N $EC_tasks_per_node -n $EC_total_tasks -j $EC_hyperthreads ./HelloMPI

Hybrid MPI-OpenMP job example

• Hybrid MPI-OpenMP jobs can be submitted to either:
  • queue np if EC_tasks_per_node x EC_threads_per_task > 18 x EC_hyperthreads
  • queue nf if EC_tasks_per_node x EC_threads_per_task <= 18 x EC_hyperthreads
• Set EC_total_tasks to the total number of MPI tasks required.
• Set EC_threads_per_task to the number OpenMP threads required. This will be the value needed for the OMP_NUM_THREADS environment variable.
• Optionally choose the amount of memory per MPI task with the EC_memory_per_task directive.
• Choose whether or not to use hyperthreading by setting EC_hyperthreads.
• The executable must be run with the aprun command
  
  

#PBS -N HelloMPI_OpenMP
#PBS -q np
#PBS -l EC_total_tasks=72
#PBS -l EC_threads_per_task=2
#PBS -l EC_hyperthreads=2

export OMP_NUM_THREADS=$EC_threads_per_task
aprun -N $EC_tasks_per_node -n $EC_total_tasks -d $OMP_NUM_THREADS -j $EC_hyperthreads ./HelloMPI_OpenMP

Fractional job example

A fractional job is a job that uses fewer than half of the total resources on a node. 

• Fractional jobs must be submitted to the nf queue
• The job must request:
  • less than 60 GBytes of memory and
  • 18 or fewer (logical) CPUs with EC_hyperthreads=1 or
  • 36 or fewer (logical) CPUs with EC_hyperthreads=2.
• Optionally set the EC_memory_per_task directive to the amount of memory per MPI task required if this is greater than the default.  

• The executable must be run with the mpiexec command.

#PBS -N HelloMPI_frac
#PBS -q nf
#PBS -l EC_total_tasks=6
#PBS -l EC_hyperthreads=1

# Load the cray-snplauncher module to add the mpiexec command to $PATH
module load cray-snplauncher

mpiexec -n $EC_total_tasks ./HelloMPI_frac

Multiple program multiple data (MPMD) job example

Multiple program multiple data (MPMD) jobs are jobs that can run either different executables which require different job geometry or else a number of instances of the same executable, possibly accessing different data. 

• MPMD jobs should be submitted to either the np or nf queue, depending on the job geometry requested
• The executable must be run with the aprun command.
  
  

#PBS -N HelloMPMD
#PBS -q np
#PBS -l EC_total_tasks=12:6
#PBS -l EC_threads_per_task=1:1
#PBS -l EC_hyperthreads=1

# Get the job geometry information into array variables for ease of use
IFS=':'
set -A tasks_per_node $EC_tasks_per_node
set -A total_tasks $EC_total_tasks

# Run model_atm.exe with 12 MPI tasks and model_ocean.exe with 6 MPI tasks
aprun -n ${total_tasks[0]} -N ${tasks_per_node[0]} ./model_atm.exe : \
      -n ${total_tasks[1]} -N ${tasks_per_node[1]} ./model_ocean.exe

#!/bin/ksh

# Put here the paths to the binaries of the MPMD run
binaries=(exec1 exec2 exec3 exec4)

# Define the number of tasks each one of them will use
ntasks=(1 1 28 20)

##########################
export OMP_NUM_THREADS=1
sum=0
i=0
for t in ${ntasks[@]}
do
    sum=$((sum + t))
    if [ $EC_FARM_ID -lt $sum ]; then
        exec ${binaries[$i]}
        break
    fi
    i=$((i+1))
done

#PBS -N PackedMPMD
#PBS -q np
#PBS -l EC_nodes=3

export PMI_NO_FORK=1
aprun -n 2 -N 2 -j 1 ./wrapper.sh : -n 48 -N 24 -j 1 ./wrapper.sh

aprun -n 72 -N 24 -j 1 ./wrapper.sh 

Further reading

1. EC_ Job Directives Summary
2. ECMWF Cray PBSpro setup concepts
3. EC_ Job Directives Quick Start
4. PBSpro at ECMWF - Cray XC30 Workshop February 2014