Storage spaces - Atos HPCF and ECS Introduction Tutorial

We will now explore the different options when it comes to storing your data.

Main filesystems

1. Connect to Atos HPCF or ECS main login node. What is your default filesystem? Can you try 4 different ways to accessing that space?

   Answer

   The default directory is your HOME directory, which is /home/$USER. It is a dedicated personal space for you, and you can always come back to that with either of the following commands:

   cd
   cd ~
   cd $HOME
   cd /home/$USER

   Your HOME directory is accessible across all Atos HPCF, ECS, VDI and EcFlow services.

2. There are 3 more main storage spaces. Create an empty file called del.me on each one of them. Then,  check that they have been created with ls, and then remove them with rm.

   Answer

   Besides HOME, you also have also access to PERM, HPCPERM and SCRATCH. Like HOME, they are all dedicated personal spaces with their corresponding environment variable. Using those environment variables over hardcoded paths is strongly recommended.

   You can use touch to create the test files:

   touch $PERM/del.me
   touch $HPCPERM/del.me
   touch $SCRATCH/del.me

   Check they exist with:

   ls -l $PERM/del.me
   ls -l $HPCPERM/del.me
   ls -l $SCRATCH/del.me

   Remove them with:

   rm $PERM/del.me
   rm $HPCPERM/del.me
   rm $SCRATCH/del.me

3. How much space have you used in each of your main 4 filesystems? How much can you store?

   Answer

   All the filesystems have quotas enforced. You can check them with the quota command

   quota

   For HOME and PERM, the snippet should look similar to:

   Quota for $HOME:
   home_b             user    1234        <space used>   <space limit>       <number of files stored>       -   *
   
   Quota for $PERM
   POSIX User      1234    <space used>   <space limit>       <number of files stored>       none

   For SCRATCH and HPCPERM the format is slightly different:

   Project quota for $SCRATCH and $SCRATCHDIR:
   Disk quotas for prj 1000001798 (pid 1000001798):
        Filesystem    used   quota   limit   grace   files   quota   limit   grace
          /ec/res4     XXX     YYY     YYY       -     ZZZ     WWW     WWW       -
   
   Project quota for $HPCPERM:
   Disk quotas for prj 2000001798 (pid 2000001798):
        Filesystem    used   quota   limit   grace   files   quota   limit   grace
          /ec/res4     XXX     YYY     YYY       -     ZZZ     WWW     WWW       -

4. If you are on the VDI, open a new terminal there. Can you access your HOME, PERM, SCRATCH and HPCPERM ?

   Answer

   HOME and PERM are NFS-based Filesystems, which are mounted on all user computing platforms at ECMWF. You may access them with $HOME and $PERM environment variables:

   ls $HOME
   ls $PERM

   However, SCRATCH and HPCPERM  are Lustre Based filesystem only available on the Atos HPCF, so  they are not available on other computing platforms such as VDI or ecFlow VMs and the corresponding environment variables are therefore not defined.

Temporary spaces

There are a number of temporary spaces you can use in your session or job.

1. Create a file called testfile on the $TMPDIR, $SCRATCHDIR and /tmp/.

   Solution

   touch $TMPDIR/testfile
   touch $SCRATCHDIR/testfile
   touch /tmp/testfile

2. Open another session in the same login node with ssh $HOSTNAME. Can you find the files you have created earlier?

   Solution

   ls -l $TMPDIR/testfile
   ls -l $SCRATCHDIR/testfile
   ls -l /tmp/testfile

   You will not see the files you created in any of those locations, since every session or job will have a different location. This includes /tmp, which is also a dedicated ramdisk for session.

Filesystem Usage

Can you decide what would be the best filesystem to use in the following cases? Why would you make that choice?

1. Store the source code, scripts and configuration of your programs and workflows

   Answer

   HOME would be the preferred choice. They are typically small but important files, so convenience of backups, snapshots and availability on all computing platforms is more important than parallel performance. 

2. Store Climate Files to be used by your model runs on Atos HPCF.

   Answer

   HPCPERM is the right choice for big files that are going to be used concurrently by parallel applications such as NWP models.

3. Working directory for your jobs.

   Answer

   SCRATCH is the go to place for your daily work. Plenty of space, good parallel performance for output data that is transient by nature. Remember to move the data you want to keep after your job somewhere else, since files not used for 30 days will be automatically deleted.

4. Store data that that you use frequently, which is considerable in size.

   Answer

   PERM if accessibility from other computing platforms or the need of snapshots is important. You can see PERM as an extension to your HOME space.

   HPCPERM, if I/O performance is more important than, especially if they are going to be used in parallel jobs on Atos HPCF.

5. Store data for longer term which is considerable in size, such as experiment results. You are not going to use it often.

   Answer

   ECFS would be the right place for longer term archival or storing backups. This is by far the place where you can store  However, data on tapes needs to be retrieved to another disk space before it can be used, so it is costly in terms of time.

   In order to use ECFS efficiently, remember to store fewer but bigger files, so it is a good idea to use tools like tar or zip to bundle together big directories with lots of files.

6. Temporary files that you don't need beyond the end of the session or job

   Answer

   $TMPDIR if performance is important and size is small, since TMPDIR is either in memory (for parallel jobs on HPCF), or on SSD disk.

   $SCRATCHDIR if size of the files is big and does not fit TMPDIR.