...
Create a new job script
naughty.shwith a time limit of 1 minutethe following contents:Code Block language bash title naughty.sh #!/bin/bash #SBATCH --mem=100 #SBATCH --output=naughty.out MEMORY=300 perl -e '"\$a="'A"'x(300$MEM*1024*1024/2);sleep sleep'60"
Submit
naughty.shto the batch system and check its status. Is it still running after one minute? Why?What happened to the job?Expand title Solution You can submit it with:
No Format sbatch naughty.sh
You can then monitor the state of your job with watch and squeue:
You can see that the job is not killed after one minute, and keeps going beyond. The reason of that is in:No Format watch -n 10 squeue -j <jobid>After a few seconds of running, you may see the job finishes and disappears. If we use sacct, we can see the job has failed, with an exit code of 9, which indicates it was killed:
No Format $ sacct -X --name naughty.sh JobID JobName QOS State ExitCode Elapsed NNodes NodeList ------------ ---------------- --------- ---------- -------- ---------- -------- -------------------- 64303470 naughty.sh ef FAILED 9:0 00:00:04 1 ac6-202Inspecting the job output the reason becomes apparent:
No Format $ grep -v ECMWF-INFO naughty.out | head -n 22 __ __ _____ __ __ _ _____ _ _ __/\_| \/ | ____| \/ | |/ /_ _| | | | __/\__ \ / |\/| | _| | |\/| | ' / | || | | | \ / /_ _\ | | | |___| | | | . \ | || |___| |__/_ _\ \/ |_| |_|_____|_| |_|_|\_\___|_____|_____|\/ BEGIN OF ECMWF MEMKILL REPORT [ERROR__ECMWF_MEMORY_SUPERVISOR:JOB_OR_SESSION_OUT_OF_MEMORy,ac6-202.bullx:/usr/local/sbin/watch_cgroup:l454:Thu Oct 26 10:49:40 2023] [summary] job/session: 64303470 requested/default memory limit for job/session: 100MiB sum of active and inactive _anonymous memory of job/session: 301MiB ACTION: about to issue: 'kill -SIGKILL' to pid: 3649110 to-be-killed process: "perl -e $a="A"x(300*1024*1024/2); sleep", with resident-segment-size: 304MiB
The job had a limit of 100 MiB, but it tried to use up to 300 MiB, so the system killed the process.Edit
naughty.shto comment the request for memory, and then play with the MEM value.Code Block language bash title naughty.sh #!/bin/bash #SBATCH --output=naughty.out ##SBATCH --mem=100 MEMORY=300 perl -e "\$a='A'x($MEM*1024*1024/2);sleep 60"How high can you with the default memory limit on the default queue before the system kills it?
Expand title Solution With trial and error, you will see the system will kill your tasks that go over 8000 MiB:
Code Block language bash title naughty.sh #!/bin/bash #SBATCH --output=naughty.out ##SBATCH --mem=100 MEMORY=8000 perl -e "\$a='A'x($MEM*1024*1024/2);sleep 60"Inspecting the job output will confirm that:
No Format $ grep -v ECMWF-INFO naughty.out | head -n 22 __ __ _____ __ __ _ _____ _ _ __/\_| \/ | ____| \/ | |/ /_ _| | | | __/\__ \ / |\/| | _| | |\/| | ' / | || | | | \ / /_ _\ | | | |___| | | | . \ | || |___| |__/_ _\ \/ |_| |_|_____|_| |_|_|\_\___|_____|_____|\/ BEGIN OF ECMWF MEMKILL REPORT [ERROR__ECMWF_MEMORY_SUPERVISOR:JOB_OR_SESSION_OUT_OF_MEMORy,ac6-202.bullx:/usr/local/sbin/watch_cgroup:l454:Thu Oct 26 11:16:43 2023] [summary] job/session: 64304303 requested/default memory limit for job/session: 8000MiB sum of active and inactive _anonymous memory of job/session: 8001MiB ACTION: about to issue: 'kill -SIGKILL' to pid: 4016899 to-be-killed process: "perl -e $a='A'x(8000*1024*1024/2); sleep", with resident-segment-size: 8004MiBHow could you have checked this beforehand instead of taking the trial and error approach?
Expand title Solution You could have checked HPC2020: Batch system, or you could also ask Slurm for this information. Default memory is defined per partition, so you can then do
No Format scontrol show partitionThe field we are looking for is
DefMemPerNode:No Format $ scontrol -o show partition | tr " " "\n" | grep -i -e "DefMem" -e "PartitionName"Can you check, without trial and error this time, what is the maximum wall clock time, maximum CPUs, and maximum memory you can request to Slurm for each QoS?
Expand title Solution Again, you will find this information ini HPC2020: Batch system, but you can also ask Slurm. These settings are part of the QoS setup so, the command is
No Format sacctmgr show qosThe field we are looking for this time are
MaxWallandMaxTRES.:No Format sacctmgr -P show qos format=name,MaxWall,MaxTRESIf you run this on HPCF, you may notice there is no maximum limit set at the QoS level for the np parallel queue, so you are bound by the maximum memory available in the node.
You can also see other limits such as the local SSD tmpdir space.
- How many jobs could you potentially have running concurrently? How many jobs could you have in the system (pending or running), before a further submission fails?