Any shell script can be submitted as a Slurm job with no modifications. In such a case, sensible default values will be applied to the job. However, you can configure the script to fit your needs through job directives. In Slurm, these are just special comments in your script, usually at the top just after the shebang line, with the form:
#SBATCH option=value
Note that these directives:
- start with the #SBATCH prefix
- are always lowercase
- have no spaces in between.
- don't expand shell variables (they are just shell comments)
A Slurm job might look like the following:
#!/bin/bash # The job name #SBATCH --job-name=helloworld # Set the error and output files #SBATCH --output=hello-%J.out #SBATCH --error=hello-%J.out # Set the initial working directory #SBATCH --chdir=/scratch/user # Choose the queue #SBATCH --qos=express # Wall clock time limit #SBATCH --time=00:05:00 # Send an email on failure #SBATCH --mail-type=FAIL # This is the job echo “Hello World!” sleep 30
You may need to change the QOS depending on the platform. Check the queues available before submitting.
You can also use these options as command line arguments to sbatch.
General directives
| Directive | Description | Default |
|---|---|---|
| Project account for resource accounting and billing purposes. | default project account for the user |
| A descriptive name of the job | Script name |
--chdir=... | Working directory of the job. The output and error files can be defined relative to this directory | submitting directory |
| Path to the file where standard output is redirected. Special placeholders for job id (%j) and the execution node (%N) | slurm-%j.out |
| Path to the file where standard error is redirected. Special placeholders for job id (%j) and the execution node (%N) | output value |
| Quality of Service (or queue) where the job is to be submitted. Check the available queues for the platform. | nf or ef |
| Wall clock limit of the job. Note that this is not cpu time limit The format can be: m, m:s, h:m:s, d-h, d-h:m or d-h:m:s | qos default time limit |
--mail-type=<type> | Notify user by email when certain event types occur. Valid values are: BEGIN, END, FAIL, REQUEUE and ALL | disabled |
--mail-user=<email> | email address to send the email | submitting user |
Directives for resource allocation
| Directive | Description | Default |
|---|---|---|
| Allocate resources for the specified number of parallel tasks. Note that a job requesting more than one must be submitted to a parallel queue. There might not be any parallel queue configured on the cluster | 1 |
| Allocate <nodes> number of nodes to the job | 1 |
| Allocate <threads> number of cpus for every task. Use for threaded applications. | 1 |
| Allocate a maximum of <tasks> tasks on every node. | node capacity |
| Allocate <threads> threads on every core (HyperThreading) | core thread capacity |
| Use or not hyperthreaded cores and define the binding accordingly. | not defined |
| Allocate <mem> memory on each node | 8 GB for serial and fractional jobs(*i, *f and *l QoS), 240 for parallel jobs (*p QoS) |
| Only on GPIL nodes (*f and *l QoS) Allocate <size> GB of SSD temporary storage | 3 GB |
See man sbatch or https://slurm.schedmd.com/sbatch.html for the complete list of directives and their options.
Job variables
Inside a job, you can benefit from some variables defined by SLURM automatically. Some examples are:
- SLURM_JOBID
- SLURM_NODELIST
SLURM_SUBMIT_DIR
For a complete list of variables defined by slurm, you submit a job which runs
env | grep SLURM
Job arrays
Job arrays offer a mechanism for submitting and managing collections of similar jobs quickly and easily. The array index values are specified using the --array or -a option of the sbatch command. The option argument can be specific array index values, a range of index values, and an optional step size as shown in the examples below. Jobs which are part of a job array will have the environment variable SLURM_ARRAY_TASK_ID set to its array index value.
# Submit a job array with index values between 0 and 31 $ sbatch --array=0-31 -N1 tmp # Submit a job array with index values of 1, 3, 5 and 7 $ sbatch --array=1,3,5,7 -N1 tmp # Submit a job array with index values between 1 and 7 # with a step size of 2 (i.e. 1, 3, 5 and 7) $ sbatch --array=1-7:2 -N1 tmp
The --array option can also be used inside the job script as a job directive. For example:
#!/bin/bash #SBATCH --job-name=my_job_array #SBATCH --array=0-31 echo “Hello World! I am task $SLURM_ARRAY_TASK_ID of the job array” sleep 30
2 Comments
Nikolaos Andritsos
For the
--workdir=working_directory, only absolute paths seem to be applicable. No variables like $HOME or $SCRATCH.Martin Leutbecher
I found the syntax
#SBATCH --array=0-31%4for array jobs quite useful. This will eventually run tasks 0 to 31 but only 4 jobs are running in parallel at any time. See
man sbatch
for details.