Any shell script can be submitted as a Slurm job with no modifications. In such a case, sensible default values will be applied to the job. However, you can configure the script to fit your needs through job directives. In Slurm, these are just special comments in your script, usually at the top just after the shebang line, with the form:

#SBATCH option=value

Note that these directives:

  • start with the #SBATCH prefix
  • are always lowercase
  • have no spaces in between.
  • don't expand shell variables (they are just shell comments)

A Slurm job might look like the following:

#!/bin/bash
# The job name
#SBATCH --job-name=helloworld
# Set the error and output files
#SBATCH --output=hello-%J.out
#SBATCH --error=hello-%J.out
# Set the initial working directory
#SBATCH --chdir=/scratch/user
# Choose the queue
#SBATCH --qos=nf
## or for ECS
#SBATCH --qos=ef
# Wall clock time limit
#SBATCH --time=00:05:00
# Send an email on failure
#SBATCH --mail-type=FAIL
# This is the job
echo “Hello World!”
sleep 30

You may need to change the QOS depending on the platform. Check the queues available before submitting.


You can also use these options as command line arguments to sbatch.

General directives

DirectiveDescriptionDefault

--account=<account>

-A <account>

Project account for resource accounting and billing purposes.default project account for the user

--job-name=<name>

-J <name>

A descriptive name of the jobScript name
--chdir=...Working directory of the job. The output and error files can be defined relative to this directorysubmitting directory

--output=<path>

-o <path>

Path to the file where standard output is redirected. Special placeholders for job id (%j) and the execution node (%N)slurm-%j.out

--error=...

-e <path>

Path to the file where standard error is redirected. Special placeholders for job id (%j) and the execution node (%N)output value

--qos=<qos>

-q <qos>

Quality of Service (or queue) where the job is to be submitted. Check the available queues for the platform.nf or ef

--time=<time>

-t <time>

Wall clock limit of the job. Note that this is not cpu time limit

The format can be: m, m:s, h:m:s, d-h, d-h:m or d-h:m:s

qos default time limit
--mail-type=<type>Notify user by email when certain event types occur. Valid values are: BEGIN, END, FAIL, REQUEUE and ALLdisabled
--mail-user=<email>email address to send the emailsubmitting user

Directives for resource allocation 


DirectiveDescriptionDefault

--ntasks=<tasks>

-n <tasks>

Allocate resources for the specified number of parallel tasks. Note that a job requesting more than one must be submitted to a parallel queue. There might not be any parallel queue configured on the cluster1

--nodes=<nodes>

-N <nodes>

Allocate <nodes> number of nodes to the job1

--cpus-per-task=<threads>

--c <threads>

Allocate <threads> number of cpus for every task. Use for threaded applications.1

--ntasks-per-node=<tasks>

Allocate a maximum of <tasks> tasks on every node.node capacity

--threads-per-core=<threads>

Allocate <threads> threads on every core (HyperThreading)core thread capacity

--hint=[no]multithread

Use or not hyperthreaded cores and define the binding accordingly.not defined

--mem=<mem>

Allocate <mem> memory on each node8 GB for serial and fractional jobs(*i, *f and *l QoS), 240 for parallel jobs (*p QoS)

--gres=ssdtmp:<size>G

Only on GPIL nodes (*f and *l QoS)

Allocate <size> GB of SSD temporary storage 

3 GB

See man sbatch or https://slurm.schedmd.com/sbatch.html for the complete list of directives and their options.

Job variables

Inside a job,  you can benefit from some variables defined by SLURM automatically. Some examples are:

  • SLURM_JOBID
  • SLURM_NODELIST
  • SLURM_SUBMIT_DIR

For a complete list of variables defined by slurm, you submit a job which runs

env | grep SLURM

Job arrays

Job arrays offer a mechanism for submitting and managing collections of similar jobs quickly and easily. The array index values are specified using the --array or -a option of the sbatch command. The option argument can be specific array index values, a range of index values, and an optional step size as shown in the examples below. Jobs which are part of a job array will have the environment variable SLURM_ARRAY_TASK_ID set to its array index value.

# Submit a job array with index values between 0 and 31
$ sbatch --array=0-31    -N1 tmp
# Submit a job array with index values of 1, 3, 5 and 7
$ sbatch --array=1,3,5,7 -N1 tmp
# Submit a job array with index values between 1 and 7
# with a step size of 2 (i.e. 1, 3, 5 and 7)
$ sbatch --array=1-7:2   -N1 tmp

The --array option can also be used inside the job script as a job directive. For example:

#!/bin/bash
#SBATCH --job-name=my_job_array
#SBATCH --array=0-31

echo “Hello World! I am task $SLURM_ARRAY_TASK_ID of the job array”
sleep 30

3 Comments

  1. For the --workdir=working_directory, only absolute paths seem to be applicable. No variables like $HOME or $SCRATCH.

  2. I found the syntax

    #SBATCH --array=0-31%4

    for array jobs quite useful. This will eventually run tasks 0 to 31 but only 4 jobs are running in parallel at any time. See

    man sbatch

    for details.

  3. #SBATCH --qos=express
     i suppose this was an old QoS
    Probably should be --qos=nf  if that is the default now