...
File System | Suitable for ... | Technology | Features | Quota | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HOME | permanent files, e. g. profile, utilities, sources, libraries, etc. | NFS | It is backed up. Snapshots available. See AG: Recovering data from snapshots Throttled I/O bandwidth from parallel compute nodes (less performance) |
| ||||||||||
| PERM | permanent files without the need for automated backups, smaller input files for serial or small processing, etc. | NFS | No backup Snapshots available. See AG: Recovering data from snapshots Throttled I/O bandwidth from parallel compute nodes (less performance) DO NOT USE IN PARALLEL APPLICATIONS DO NOT USE FOR JOB STANDARD OUTPUT/ERROR |
| ||||||||||
| HPCPERM | permanent files without the need for automated backups, bigger input files for parallel model runs, climate files, std output, etc. | Lustre | No Backup No snapshots No automatic deletion |
| ||||||||||
| SCRATCH | all temporary (large) files. Main storage for your jobs and experiments input and output files. | Lustre | Automatic deletion after 30 days of last access - implemented since 27 March 2023 No snapshots No backup |
| ||||||||||
SCRATCHDIR | Big temporary data for an individual session or job, not as fast as TMPDIR but higher capacity. Files accessible from all cluster. | Lustre | Deleted at the end of session or job Created per session/ job as a subdirectory in SCRATCH | part of SCRATCH quota | ||||||||||
TMPDIR | Fast temporary data for an individual session or job, small files only. Local to every node. | SSD on shared (GPIL) nodes (*f QoSs) | NVME | Deleted at the end of session or job Created | per session/ jobOn ECS: 3 GBper session/ job by default. | On HPC:
To request more space in your jobs you can use the Slurm directive:
For ecinteractive and Jupyterhub sessions, space and limits are shared with LOCALSSD | ||||||||
RAM on exclusive parallel compute and GPU nodes (*p and *g QoSs) | no limit (maximum memory of the node) | |||||||||||||
LOCALSSD | Fast, local, non-critical data and files used in ecinteractive and JupyterHub sessions. Its contents are automatically archived when the session finishes so users can restore them on their next session and carry on where they left off. Can be used for development/compilation of projects interactively. See AG: Local SSD storage for interactive sessions for more information on how to use this feature. | SSD on shared (GPIL) nodes for ecinteractive and JupyterHub sessions only. (ni QoS) GPU nodes are excluded | NVME | Archived automatically at the end of session or job into Recover manually on next session with:
On ECS:
On HPC: 3 GB per session/job by default. | Space and limits shared with TMPDIR |
...
| Tip | ||
|---|---|---|
| ||
Those filesystems can be conveniently referenced from your session and scripts using the environment variables of the same name: $HOME, $PERM, $HPCPERM, $SCRATCH, $SCRATCHDIR, $TMPDIR and $LOCALSSD. $TEMP, which in the past was an alias to $SCRATCH, has been deprecated and is no longer defined. Please use $SCRATCH instead |
| Tip | ||||
|---|---|---|---|---|
| ||||
When running on the shared GPIL nodes (*f and *i QoSs), you may request a bigger space in the SSD-backed TMPDIR with the extra SBATCH option:
With <size> being a number up to 20 GB on ECS and 100 GB on HPCF. If that is still not enough for you, you may point your TMPDIR to SCRATCHDIR:
Note that on interactive, non-gpu sessions using ecinteractive and Jupyterhub, the local space on disk is shared between TMPDIR and LOCALSSD. |
...
Some special directories are not disk-based but actually mapped into the node's main memory. There are no limits set on exclusive nodes running on parallel queues. However, when running on shared nodes (GPILs) on fractional or interactive queues, this could lead to a single application or user exhausting all the memory of the node and thus impacting others. This is why the following limits are set:They are unique for every session, and are limited by the memory resources requested in the job.
| Directory | |
|---|---|
| /tmp | |
| /var/tmp | |
| /dev/shm | |
| Directory | New limit |
| /tmp | 428 GB (80% of available memory) per user's session |
| /var/tmp | 428 GB (80% of available memory) per user's session |
| /dev/shm | 428 GB (80% of available memory) per user's session |
| $XDG_RUNTIME_DIR | 64 MB per user's session |
Users should instead use the general purpose file systems available, and in particular, $TMPDIR or $SCRATCHDIR for temporary storage per session or job.
| Note | ||
|---|---|---|
| ||
Any data left on those spaces will be automatically deleted at the end of the job or session. |
Special Filesystems
See hereafter a list of the specialised filesystems on Atos:
...
. |
| Show If | ||
|---|---|---|
| ||
Project FilesystemsCertain projects with special requirements have dedicated filesystems or volumes that are automatically mounted under Those project volumes share the same backend as PERM, so the similar features apply:
The quota command will not show the limits for those filesystems. If you have any queries about them, please do raise an issue via our ECMWF Support Portal. |
...