ECFS is ECMWF's File Storage system. ECFS is a file oriented client-server application, providing facilities to archive and retrieve files between your local workstation or server HPC/ECS and the Data Handling System (DHS), which is based on HPSS (High Performance Storage System). The system is non-transparent: you must issue explicit commands to store files into ECFS, or to retrieve them back into local (client) storage.
...
- DON'T archive many small files separately. ECFS is most efficient at handling a small number of large files. Thus
- DO tar and compress (or gzip) many small files into one large file when archiving (say) a directory or use the
ecfsdircommand. - DON'T copy in/out the same files frequently.
- If you wish to archive files for a short period only (less than 90 days) DO store them in the
ectmp:domain. Then they will be automatically deleted after 90 days without any further action from you. Check the existence of a local copy before getting the ECFS version of a file:
#!/bin/ksh if [ ! -r $SCRATCH/myfile ] then ecp ec:myfile $SCRATCH/. fi
- If it is required to store a large number ( > 5000) of files into ECFS then DO contact the Service Desk ECMWF Support Portal in advance to discuss the most efficient way to store/retrieve this large number.
...