ECFS is ECMWF ECMWF's File Storage File Storage system. ECFS is a file oriented client-server application, providing facilities to store archive and retrieve entire files between your local workstation or server and the Data Handling System (DHS), which is based on HPSS (High Performance Storage System). The system is non-transparent: you must issue explicit commands to store files into ECFS, or to retrieve them back into local (client) storage.

...

The domain names shown above (ec:, ectmp:) are used in the various ECFS commands to indicate which domain you are working with.

Note that, as an alternative, the ectmp: domain can be referenced by ec:/TMP. Thus ectmp:/uid/newdir and ec:/TMP/uid/newdir are equivalent:

...

ECFS supports the transport of files between various clients and the ECFS storage system. However, the transport of files between clients is not supported.

ECFS is available on the AIX, Linux and other platforms, in both the C-shell and Kornand supports C- , Korn- and bash-shell environments.

Files transferred between a client and ECFS storage can be up to 32 GB in size.

...

eumask - Change the current ECFS umask

Backup support

By default files saved in ECFS will not have a backup copy created.

...

The existence of backup copies will be indicated by the first character of the els -l output:

- for files with no secondary copy;
b for files with a secondary copy.

e.g.

 br--r----- 1 uid group 510 Nov 19 2012 essential_data
 -r--r----- 1 uid group 510 Nov 19 2012 unimportant_data

NOTE: Files are removed from ECFS with a soft-delete: files will still be kept for some time (currently 30 days) during which it will be possible, on request, to un-delete any file that was removed/deleted by mistake. After that period any removal will become permanent, irrespective of the existence of backup copies. 

Archive usage monitoring

The ecfs_status command will give you the most recent status on ECFS usage for your default project account.

To get an overview on their your ECFS usage, users you can also refer to the audit files ec:ecfs_audit and/or ectmp:ecfs_audit.tmp which are created once per month and contain a complete list of a user's your files in each ECFS domain.

...

set +e
ecp nofile ec:
RC=$?
set -e
if [ $RC -gt 0 ]
  then
  echo " ECFS call exited with RC= $RC"
fi

DOs and DON'Ts

...

• ECFS is most efficient at handling a small number of large files. Thus DO tar and compress (or gzip) many small files into one large file when archiving (say) a directory or use the ecfsdir command.
  DON'T archive many small files separately.
• Do not DON'T copy in/out the same files frequently.
• If you wish to archive files for a short period only (less than 90 days) DO store them in the ectmp: domain. Then they will be automatically deleted after 90 days without any further action from you.

• Check the existence of a local copy before getting the ECFS version of a file:

  #!/bin/ksh if [ ! -r $SCRATCH/myfile ] then ecp ec:myfile $SCRATCH/. fi

  #!/bin/csh if ( ! -r $SCRATCH/myfile ) then ecp ec:myfile $SCRATCH/. endif

• If it is required to store a large number ( > 5000) of files into ECFS then DO contact User Support in advance to discuss the most efficient way to store/retrieve this large number.

...