...
Note that running the command multiple times will always overwrite the file on destination.
Advanced: High Performance Transfers with bbcp
You may also use a specialised tool called bbcp for best transfer rates. It is available on Atos HPCF. This tool is not as flexible as rsync when it comes to updating existing or partial copies, but it should be quicker when doing a one-off transfer.
For example, if you wanted to transfer a directory called mydataset in your SCRATCH in Reading, you could initiate the transfer from ECGATE:
| No Format |
|---|
user@yourlaptop:~> bbcp -rp -s 10 -P 2 mydataset hpc-login:/scratch/user/ |
The options above would perform a recursive copy, preserving file permissions and using 10 parallel streams. It would also report progress every 2 seconds.
You may find all the details in the bbcp official documentation.
| Note | ||
|---|---|---|
| ||
bbcp will not copy soft links, so if you are copying an entire directory structure that contains them, you may need to copy them over at a second stage with another tool like rsync. |
...
Transferring files between your computer and the Atos HPCF or ECFS domains via ECaccess gateway
For a data transfer from an Internet site to ECMWF, invoke a connection to the ECaccess gateway in Bologna:
| No Format |
|---|
user@yourlaptop:~> sftp user@boaccess.ecmwf.int
sftp> cd ECSCRATCH
sftp> put MyFile |
Advanced: High Performance Transfers with bbcp
You may also use a specialised tool called bbcp for best transfer rates. It is available on Atos HPCF. This tool is not as flexible as rsync when it comes to updating existing or partial copies, but it should be quicker when doing a one-off transfer.
For example, if you wanted to transfer a directory called mydataset in your SCRATCH in Reading, you could initiate the transfer from ECGATE:
| No Format |
|---|
user@yourlaptop:~> bbcp -rp -s 10 -P 2 mydataset hpc-login:/scratch/user/ |
The options above would perform a recursive copy, preserving file permissions and using 10 parallel streams. It would also report progress every 2 seconds.
You may find all the details in the bbcp official documentation.
| Note | ||
|---|---|---|
| ||
bbcp will not copy soft links, so if you are copying an entire directory structure that contains them, you may need to copy them over at a second stage with another tool like rsync. |
Fetching data from ECMWF to your computer
Transferring a directory tree between the Atos HPCF and your computer with rsync
This is the recommended and most versatile option. You may initiate the transfer from your computer for the standard set of filesystems:
| No Format |
|---|
user@yourlaptop:~> rsync -avz hpc-login:/scratch/user/mydataset . |
This command can be run multiple times, since only new or modified files will be transferred.
| Tip |
|---|
You may add the |
Transferring a directory tree between the Atos HPCF and your computer with
...
scp
This is the recommended and most versatile option. You may initiate the transfer from your computer for the standard set of filesystems:
| No Format |
|---|
user@yourlaptop:~> rsync -avzscp hpc-login:/scratch/user/mydatasetmyfile . |
This command can be run multiple times, since only new or modified files will be transferred.
...
Note that running the command multiple times will always overwrite the file on destination.
Transferring a directory tree between the Atos HPCF and your computer with
...
sftp
You may initiate the transfer from your computer:
| No Format |
|---|
user@yourlaptop:~> scpsftp hpc-login: sftp> cd /scratch/user/myfile . sftp> get myfile |
Note that running the command multiple times will always overwrite the file on destination.
Transferring
...
files between the Atos HPCF
...
or ECFS domains to your computer via ECaccess gateway
For a data transfer from an Internet site to ECMWF, invoke a connection to the ECaccess gateway in BolognaYou may initiate the transfer from your computer:
| No Format |
|---|
user@yourlaptop:~> sftp hpc-loginuser@boaccess.ecmwf.int sftp> cd ECHOST/hpc-login/scratchhpcperm/user sftp> get myfile |
...
MyFile |
Pushing data from ECMWF to external sites
...