Versions Compared

Old Version 3

changes.mady.by.user Xavier Abellan

Saved on

compared with

New Version Current

changes.mady.by.user Daniel Varela Santoalla

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Store the source code, scripts and configuration of your programs and workflows

    Expand
    titleAnswer

    HOME would be the preferred choice. They are typically small but important files, so convenience of backups, snapshots and availability on all computing platforms is more important than parallel performance. 


  2. Store Climate Files to be used by your model runs on Atos HPCF.

    Expand
    titleAnswer

    HPCPERM is the right choice for big files that are going to be used concurrently by parallel applications such as NWP models.


  3. Working directory for your jobs.

    Expand
    titleAnswer

    SCRATCH is the go to place for your daily work. Plenty of space, good parallel performance for output data that is transient by nature. Remember to move the data you want to keep after your job somewhere else, since files not used for 30 days will be automatically deleted.


  4. Store data that that you use frequently, which is considerable in size.

    Expand
    titleAnswer

    PERM if accessibility from other computing platforms or the need of snapshots is important. You can see PERM as an extension to your HOME space.

    HPCPERM, if I/O performance is more important than, especially if they are going to be used in parallel jobs on Atos HPCF.


  5. Store data for longer term which is considerable in size, such as experiment results. You are not going to use it often.

    Expand
    titleAnswer

    ECFS would be the right place for longer term archival or storing backups. This is by far the place where you can store  However, data on tapes needs to be retrieved to another disk space before it can be used, so it is costly in terms of time.

    In order to use ECFS efficiently, remember to store fewer but bigger files, so it is a good idea to use tools like tar or zip to bundle together big directories with lots of files.


  6. Temporary files that you don't need beyond the end of the session or job

    Expand
    titleAnswer

    $TMPDIR if performance is important and size is small, since TMPDIR is either in memory (for parallel jobs on HPCF), or on SSD disk.

    $SCRATCHDIR if size of the files is big and does not fit TMPDIR.

Recovering Deleted files

Info
titleReference documentation

HPC2020: Filesystems

Imagine you have accidentally deleted ~/.profile in your HOME directory. Can you get back the latest version?

Expand
titleSolution

You can use the snapshots . You can list all the versions available with:

No Format
ls -l ~/.snapshot/*/.profile

To recover, you would just need to copy the file back into place. 

For longer time spans, use the utility home_snap to get the locations


Include Page
How to recover deleted files in the Atos HPC and ECS service filesystems and storage
How to recover deleted files in the Atos HPC and ECS service filesystems and storage

Imagine you have accidentally deleted a file in your PERM directory. Can you get back the latest version?

Expand
titleSolution

You can use the snapshots . You can list all the versions available with:

No Format
perm_snap

Note that the snapshots are less frequent in PERM.

Imagine you have accidentally deleted a file in your SCRATCH or HPCPERM directories. Can you get back the latest version?

...

titleSolution

...