...
ECFS files are stored in domains. Currently two domains are available to users.
| ec: | The permanent domain where files are stored indefinitely. This is the default domain. |
| ectmp: | The temporary domain where files are stored for 90 days, after which they are automatically deleted. Note that once a file has been deleted it CANNOT be recovered. |
The domain names shown above (ec:, ectmp:) are used in the various ECFS commands to indicate which domain you are working with.
...
ECaccess allows users to transfer files between their remote host and an ECFS domain at ECMWF.
| Table of Contents |
|---|
ECFS commands
The Unix style of file interface has been adopted by ECFS:
...
All available ECFS commands are described in more detail in the relevant man pages:
ecat | Copy files between a domain and STDIN/STDOUT |
ecd | Change the current ECFS working directory |
ecfs_status | Get status on ECFS usage (on server only) |
ecfsdir | Archive or retrieve a complete UNIX directory as one ECFS file |
echgrp | Change the group ownership of an ECFS file or directory |
echmod | Change the permissions of a ECFS file or directory |
ecp | Copy files across ECFS domains, including UNIX |
els | List ECFS files |
emkdir | Create empty ECFS directories |
emove | Rename files or directories within an ECFS domain |
emv | Move files across ECFS domains, including UNIX |
epwd | Display the current ECFS working directory for the relevant domain |
ermdir | Remove empty ECFS directories |
erm | Remove ECFS files |
etest | Check file types and compare file attributes |
etouch | Change file timestamps |
eumask | Change the current ECFS umask |
File name 'globbing'
File name globbing refers to the ability of Unix shells to allow users to specify a single pattern that expands to a list of file names that match that pattern. The typical case is the '*' character: A specification of, for instance, file* will expand to a list of all local files whose names, such as file1 or file_X, match that pattern.
...
- for files with no secondary copy;
b for files with a secondary copy.
e.g.
br--r----- 1 uid group 510 Nov 19 2012 essential_data
-r--r----- 1 uid group 510 Nov 19 2012 unimportant_data
|
NOTE: Files are removed from ECFS with a soft-delete: files will still be kept for some time (currently 30 days) during which it will be possible, on request, to un-delete any file that was removed/deleted by mistake. After that period any removal will become permanent, irrespective of the existence of backup copies.
...
Set a trap for the entire script:
trap " echo ECFS call exited with RC= \$? " ERR
|
or catch any errors on each call:
set +e
ecp nofile ec:
RC=$?
set -e
if [ $RC -gt 0 ]
then
echo " ECFS call exited with RC= $RC"
fi
|
DOs and DON'Ts
- DON'T archive many small files separately. ECFS is most efficient at handling a small number of large files. Thus
- Thus DO tar and compress (or gzip) many small files into one large file when archiving (say) a directory or use the
ecfsdircommand. - DON'T archive many small files separately.DON'T copy in/out the same files frequently.
- If you wish to archive files for a short period only (less than 90 days) DO store them in the
ectmp:domain. Then they will be automatically deleted after 90 days without any further action from you. Check the existence of a local copy before getting the ECFS version of a file:
#!/bin/csh if ( ! -r $SCRATCH/myfile ) then ecp ec:myfile $SCRATCH/. endif#!/bin/ksh if [ ! -r $SCRATCH/myfile ] then ecp ec:myfile $SCRATCH/. fi- If it is required to store a large number ( > 5000) of files into ECFS then DO contact User Support in advance to discuss the most efficient way to store/retrieve this large number.
...