Versions Compared

Old Version 7

changes.mady.by.user Paul Dando

Saved on

compared with

New Version Current

changes.mady.by.user Xavier Abellan

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Job Directives

Any shell script can be submitted as a Slurm job with no modifications. In such a case, sensible default values will be applied to the job. However, you can configure the script to fit your needs through job directives. In Slurm, these are just special comments in your script, usually at the top just after the shebang line, with the form:

...

Code Block
languagebash
#!/bin/bash
# The job name
#SBATCH --job-name=helloworld
# Set the error and output files
#SBATCH --output=hello-%J.out
#SBATCH --error=hello-%J.out
# Set the initial working directory
#SBATCH --workdirchdir=/scratch/us/usxauser
# Choose the queue
#SBATCH -–qos-qos=express
# Wall clock time limit
#SBATCH --time=00:05:00
# Send an email on failure
#SBATCH --mail-type=FAIL
# This is the job
echo “Hello World!”
sleep 30

...


 Note
You may need to change the QOS depending on the platform. Check the queues available before submitting.

 Info
You can also use these options as command line arguments to sbatch.
General directives
DirectiveDescriptionDefault
--account=<account>
-A <account>
Project account for resource accounting and billing purposes.default project account for the user
--job-name=
...
<name>
-J <name>
A descriptive name of the jobScript name
--
output
chdir=...
      
Working directory of the job. The output and error files can be defined relative to this directorysubmitting directory
--output=<path>
-o <path>
Path to the file where standard output is redirected. Special placeholders for job id (%j) and the execution node (%N)slurm-%j.out
--error=...
-e <path>
Path to the file where standard error is redirected. Special placeholders for job id (%j) and the execution node (%N)output value
--
workdir
qos=
...
<qos>
-q <qos>
Working directory of the job. The output and error files can be defined relative to this directorysubmitting directory--qos=...
Quality of Service (or queue) where the job is to be submitted
normal
. Check the available queues for the platform.nf or ef
--time=
...
<time>
-t <time>
Wall clock limit of the job. Note that this is not cpu time limit
The format can be: m, m:s, h:m:s, d-h, d-h:m or d-h:m:s
qos default time limit
--mail-type=
...
<type>Notify user by email when certain event types occur. Valid values are: BEGIN, END, FAIL, REQUEUE and ALLdisabled
--mail-user=
...
<email>email address to send the emailsubmitting user
Directives for resource allocation 

DirectiveDescriptionDefault
--ntasks=
..
<tasks>
-n <tasks>
Allocate resources for the specified number of parallel tasks. Note that a job requesting more than one must be submitted to a parallel queue. There might not be any parallel queue configured on the cluster1
--nodes=<nodes>
-N <nodes>
Allocate <nodes> number of nodes to the job1
--cpus-per-task=<threads>
--c <threads>
Allocate <threads> number of cpus for every task. Use for threaded applications.1
--ntasks-per-node=<tasks>
Allocate a maximum of <tasks> tasks on every node.node capacity
--threads-per-core=<threads>
Allocate <threads> threads on every core (HyperThreading)core thread capacity
--hint=[no]multithread
Use or not hyperthreaded cores and define the binding accordingly.not defined
--mem=<mem>
Allocate <mem> memory on each node8 GB for serial and fractional jobs(*i, *f and *l QoS), 240 for parallel jobs (*p QoS)
--gres=ssdtmp:<size>G
Only on GPIL nodes (*f and *l QoS)
Allocate <size> GB of SSD temporary storage 
3 GB

 Tip
See man sbatch or https://slurm.schedmd.com/sbatch.html for the complete list of directives and their options.
 Info
You can also use these options as command line arguments to sbatch.
Job variables
Inside a job,  you can benefit from some variables defined by SLURM automatically. Some examples are:
  • SLURM_JOBID
  • SLURM_NODELIST
  • SLURM_SUBMIT_DIR
 Tip
Job arrays
...
For a complete list of variables defined by slurm, you submit a job which runs
 No Format
env | grep SLURM

Job arrays
...
Job arrays offer a mechanism for submitting and managing collections of similar jobs quickly and easily. The array index values are specified using the --array or -a option of the sbatch command. The option argument can be specific array index values, a range of index values, and an optional step size as shown in the examples below. Jobs which are part of a job array will have the environment variable SLURM_ARRAY_TASK_ID set to its array index value.
...
 Tip
The --array option can also be used inside the job script as a job directive. For example:
 Code Block
languagebash
#!/bin/bash
#SBATCH --job-name=my_job_array
#SBATCH --array=0-31

echo “Hello World! I am task $SLURM_ARRAY_TASK_ID of the job array”
sleep 30
 Show If
groupecmwf
Job arrays or other multiple concurrent jobs using IDL
If you are running a job array or other multiple concurrent jobs on lxc that call IDL then it is good to constrain these to run on a small number of nodes to limit the number of IDL licences requested.  To do this add the --constraint=idl option to the scripts job directives:
 Code Block
languagebash
#!/bin/bash
#SBATCH --job-name=my_idl_job_array
#SBATCH --array=0-31
#SBATCH --constraint=idl

echo “Hello World! I am task $SLURM_ARRAY_TASK_ID of the job array”
module load idl
idl << EOF
.run my_idl_program.pro
my_idl_program
EOF
sleep 30