...

Logging into Atos HPCF and ECS

First of all, let's try to connect to the computing services via SSH:

1. Access the default login node of the ATOS HPCF or ECS and take note of what node you are in 

   Expand
   title Solution - HPCF
   No Format ssh hpc-login hostname

   
   

   Expand
   title Solution - ECS
   No Format ssh ecs-login hostname

   
   

2. Open a new tab in your terminal and connect again. Did you get the same hostname? Why is that?

   Expand
   title Answer
   hpc-login hostname is an alias to a load-balanced service of login nodes. You may land on a different one every time you connect. ecs-login is an alias to a specific login node of the ECS virtual cluster, which may be changed. It is not automatically load-balanced, so you will typically land on the same node on consecutive connections. Both aliases will always point to a working login node, and the actual node and complex behind it may change depending on the load, system sessions or outages.

   
   

3. Now, from your open SSH session on Atos HPCF or ECS, connect to the main login alias again. Did it ask for a password? Can you set your account up so jumps between hosts are done without a password?

   Expand
   title Answer
   Password-less SSH between ECMWF hosts such as Atos HPCF or ECS nodes, or VDI hosts is not set up by default. If you were asked for a password, you can run the following command from your Atos HPCF, ECS or VDI session to set up  key-based authentication: No Format ssh-key-setup After this you should be able to jump between hosts without having to introduce your password. Besides being convenient, this setup is also necessary for other tools such as ECACCESS or ecinteractive to work properly.

   
   

Storage spaces

We will now explore the different options when it comes to storing your data.

Main filesystems

1. Connect to Atos HPCF or ECS main login node. What is your default filesystem? Can you try 4 different ways to accessing that space?

   Expand
   title Answer
   The default directory is your HOME directory, which is /home/$USER. It is a dedicated personal space for you, and you can always come back to that with either of the following commands: No Format cd cd ~ cd $HOME cd /home/$USER Your HOME directory is accessible across all Atos HPCF, ECS, VDI and EcFlow services.

   
   

2. There are 3 more main storage spaces. Create an empty file called del.me on each one of them? Check that they have been created with ls, and then remove them with rm.

   Expand
   title Answer
   Besides HOME, you also have also access to PERM, HPCPERM and SCRATCH. Like HOME, they are all dedicated personal spaces with their corresponding environment variable. Using those environment variables over hardcoded paths is strongly recommended. You can use touch to create the test files: No Format touch $PERM/del.me touch $HPCPERM/del.me touch $SCRATCH/del.me Check they exist with: No Format ls -l $PERM/del.me ls -l $HPCPERM/del.me ls -l $SCRATCH/del.me Remove them with: No Format rm $PERM/del.me rm $HPCPERM/del.me rm $SCRATCH/del.me

   
   

3. How much space have you used in each of your main 4 filesystems? How much can you store?

   Expand
   title Answer
   All the filesystems have quotas enforced. You can check them with the quota command No Format quota For HOME and PERM, the snippet should look similar to: No Format Quota for $HOME: home_b             user    1234        <space used>   <space limit>       <number of files stored>       -   * Quota for $PERM POSIX User      1234    <space used>   <space limit>       <number of files stored>       none For SCRATCH and HPCPERM the format is slightly different: No Format Project quota for $SCRATCH and $SCRATCHDIR: Disk quotas for prj 1000001798 (pid 1000001798): Filesystem used quota limit grace files quota limit grace /ec/res4 XXX YYY YYY - ZZZ WWW WWW - Project quota for $HPCPERM: Disk quotas for prj 2000001798 (pid 2000001798): Filesystem used quota limit grace files quota limit grace /ec/res4     XXX YYY YYY - ZZZ WWW WWW -

   
   

4. Can you decide what would be the best filesystem to use in the following cases? Why would you make that choice?
   1. Store the source code, scripts and configuration of your programs and workflows
   2. Store Climate Files to be used by your model runs on Atos HPCF
   3. Working directory for your jobs
   4. Store data that that you use frequently, which is considerable in size.

5. If you are on the VDI, open a new terminal there. Can you access your HOME, PERM, SCRATCH and HPCPERM ?

   Expand
   title Answer
   HOME and PERM are NFS-based Filesystems, which are mounted on all user computing platforms at ECMWF. You may access them with $HOME and $PERM environment variables: No Format ls $HOME ls $PERM However, SCRATCH and HPCPERM  are Lustre Based filesystem only available on the Atos HPCF, so  they are not available on other computing platforms such as VDI or ecFlow VMs and the corresponding environment variables are therefore not defined.

   
   

Temporary spaces

There are