Top tips when working with SLURM

  • Put all your SLURM directives at the top of the script file, above any commands. Any directive after an executable line in the script is ignored. 
  • Note that you can pass SLURM directives as options to the sbatch command.
  • Check man sbatch for a complete reference of all the options and their description.

Main command line tools

The table summarises the main Slurm commands and their PBSpro equivalents.

User commands

PBSpro

Slurm
Job submissionqsub [<pbs_options>] <job_script>sbatch [<sbatch_options>] <job_script>
Job cancellationqdel <job_id>scancel <job_id>
Job statusqscan [-u <uid>] [<job_id>]squeue [-u <uid>] -j [<job_id>]
Queue informationqstat -Q [-f] [<queue>]squeue [<squeue_options>]

Main directives

This table shows the main directives used in both Slurm and PBSpro side by side

KeywordPBSproSlurmNotes
Prefix#PBS#SBATCH
Queue-q <queue>

--qos=<queue> 

-q <queue>


Job Name

-N <job_name>

-l EC_job_name=<job_name>

--job-name=<job_name>

-J <job_name>


Shell-S /usr/bin/ksh#!/usr/bin/kshSlurm uses the shebang of the script as the shell to interpret it.
Wall-clock limit-l walltime=<hh:mm:ss>

--time=<hh:mm:ss>

-t <hh:mm:ss>


Initial working directoryno equivalent

--chdir=<path>

-D <path>


Job output-o <output_file>

--output=<output_file>

-o <output_file>


Job error-e <error_file>

--error=<error_file>

-e <error_file>

Email notification-m <event>--mail-type=<type>

Type can be NONE, BEGIN, END, FAIL, REQUEUE, ALL

Email user-M <email>--mail-user=<email>
Set environment variables-v <ENV1>=<value1>, <ENV2>=<value2>--export=<ENV1>=<value1>,<ENV2>=<value2>
Copy environment-V--export=ALLThe default on this platform is to copy no environment i.e. start a fresh session.
Number of nodes-l EC_nodes=<nodes>

--nodes=<nodes>

-N <nodes>


Total number of MPI tasks-l EC_total_tasks=<tasks>

--ntasks=<tasks>

-n <tasks>


Number of parallel threads per MPI task-l EC_threads_per_task=<threads>

--cpus-per-task=<threads>

--c <threads>


Number of MPI tasks per node-l EC_tasks_per_node=<tasks>

--ntasks-per-node=<tasks>



Consumable Memory per MPI task-l EC_memory_per_task=<memory>--mem-per-cpu=<memory>
Use hyperthreading / SMT-l EC_hyperthreads=2 / 1 --threads-per-core=<threads>
Billing account-l EC_billing_account=<account>

--account=<account>

-A <account>


TMPDIR size-l EC_job_tmpdir=<size>G

--gres=ssdtmp:<size>G


Only relevant to shared nodes (*f) QoSs.

TMPDIR is memory based on Cray, but SSD-backed on Atos