ECFS is is ECMWF's File Storage system. ECFS is a file oriented client-server application, providing facilities to archive and retrieve files between your local workstation or server HPC/ECS and the Data Handling System (DHS), which is based on HPSS (High Performance Storage System). The system is non-transparent: you must issue explicit commands to store files into ECFS, or to retrieve them back into local (client) storage.
...
ECFS supports the transport of files between various clients and the ECFS storage system. However, the transport of files between clients is not supported.
ECFS is available on AIX, Linux and other platformsthe Atos HPCF and ECS, and supports C- , the Korn- and bash-shell environments.
...
ecd ec:<ecfs_path_name> cat>sourcelist<<eof ecp --order=tape -F sourcelist <local <local_directory> |
|---|
Note that you have to use the '-F <file_name>' option.
If some files in the subdirectories listed in the sourcelist ' optionfile have the same name, the ecp overwriting options (-o, -n, -e, -u) will then also be adhered to. This means that you will typically end up with one single file, either the first one on tape (default option -n) or the last one on tape (option -o). To transfer all the files with the same name, you should use the option '–parents' of ecp, which will created the different local subdirectories corresponding to the ECFS directories.
Error handling
The following techniques are suggested for trapping ECFS error codes when running batch scripts in the Korn shell environment:
...
- DON'T archive many small files separately. ECFS is most efficient at handling a small number of large files. Thus
- DO tar and compress (or gzip) many small files into one large file when archiving (say) a directory or use the
ecfsdircommand. - DON'T copy in/out the same files frequently.
- If you wish to archive files for a short period only (less than 90 days) DO store them in the
ectmp:domain. Then they will be automatically deleted after 90 days without any further action from you. Check the existence of a local copy before getting the ECFS version of a file:
#!/bin/ksh if [ ! -r $SCRATCH/myfile ] then ecp ec:myfile $SCRATCH/. fi
- If it is required to store a large number ( > 5000) of files into ECFS then DO contact the Service Desk ECMWF Support Portal in advance to discuss the most efficient way to store/retrieve this large number.
...