Question
I'm handling large data files and/or performing computations that require more memory or disk space than I have available on Atos - what can I do?
Answer
Debug info
First, make sure you can see Metview's error messages. If running a macro, or using the graphical user interface, make sure to start Metview with the -slog option (sends all output to standard output). If running a Python script, set environment variable METVIEW_PYTHON_DEBUG=1 before starting Python or add this line before importing the 'metview' module:
Code Block |
---|
os.environ["METVIEW_PYTHON_DEBUG"] = "1" |
Temporary disk space
If you see an error message similar to the following, please see A Metview session on Atos has run out of tmp space - Metview FAQ
Code Block |
---|
/etc/ecmwf/ssd/ssd1/tmpdirs/xxx.34391055/mv.211486.xxx/config: No space left on device |
GRIB field larger than default buffer size
If you see an error message similar to the following:
...
Do this before starting Metview, or before importing the 'metview' module (see the example for setting METVIEW_PYTHON_DEBUG, above).
Process running out of memory
If you get a memkill, you have some options to give more memory to your process.
ensure that your ecinteractive shell has enough memory (current limit of 32GB), e.g.
Code Block ecinteractive -c8 -m32GB -s8GB
- run your job in batch - this involves writing a short shell script in which you specify the amount of memory your job needs, e.g.
Code Block title myjob.sh #!/bin/bash #SBATCH --mem-per-cpu=64G module load ecmwf-toolbox/new metview -slog -b mytask.mv
Then use 'sbatch' to run this script:
Code Block sbatch ./myjob.sh
Related articles
Content by Label | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...