...
| Show If | ||
|---|---|---|
| ||
Transferring files from local Reading-based workstation disks /var/tmp or /hugetmpDirect transfers from your workstation to the spaces in Bologna are discouraged, since they would not use the dedicated network link between the two data centres and transfer rates will be poor. Instead, you may force the transfers to jump through ecgate for best results. For example, to transfer a directory and its contents under your local disk onto the PERM space in Bologna:
Transferring files from Reading-based workstation filesystems $HOME and $PERMThese fileystems are also available in ecgate and lxc, so we would recommend users to run the transfers from either of those systems and not directly from the workstation. See the example above. |
Advanced: High Performance Transfers with bbcp
You may also use a specialised tool called bbcp for best transfer rates. It is available on Atos HPCF, as well as Reading Based platforms Cray HPCF and ECGATE. This tool is not as flexible as rsync when it comes to updating existing or partial copies, but it should be quicker when doing a one-off transfer.
For example, if you wanted to transfer a directory called mydataset in your SCRATCH in Reading, you could initiate the transfer from ECGATE:
| No Format |
|---|
user@ecgb11:~> bbcp -rp -s 10 -P 2 $SCRATCH/mydataset hpc-login:/scratch/user/ |
or alternatively from Atos HPCF:
| No Format |
|---|
user@aa6-100:~> bbcp -rp -s 10 -P 2 ecgate:/scratch/group/user/mydataset $SCRATCH/ |
The options above would perform a recursive copy, preserving file permissions and using 10 parallel streams. It would also report progress every 2 seconds.
You may find all the details in the bbcp official documentation.
| Note | ||
|---|---|---|
| ||
bbcp will not copy soft links, so if you are copying an entire directory structure that contains them, you may need to copy them over at a second stage with another tool like rsync. |