For transfers, we recommend using rsync which will transfer the files over an ssh connection. For that, you will need to have Teleport configured with the apropriate settings in your ssh config file.
Any file transfer tool that supports SSH and the ProxyJump feature should work, such as the command line tools sftp or scp. Alternatively, you may also use the Linux Virtual Desktop and its folder sharing capabilites to copy local files to your ECMWF's HOME or PERM.
All examples are done with the generic HPCF login node hpc-login, but you should use ecs-login if you don't have access to the full HPCF service.
Transferring a directory tree between your computer and the Atos HPCF with rsync
This is the recommended and most versatile option. You may initiate the transfer from your computer for the standard set of filesystems:
user@yourlaptop:~> rsync -avz mydataset hpc-login:/scratch/user/
This command can be run multiple times, since only new or modified files will be transferred.
You may add the --delete option if you also wish to delete files on the destination that have been removed from the source.
Transferring a single file between your computer and the Atos HPCF with scp
You may initiate the transfer from your computer:
user@yourlaptop:~> scp myfile hpc-login:/scratch/user/
Note that running the command multiple times will always overwrite the file on destination.
Transferring a single file between your computer and the Atos HPCF with sftp
You may initiate the transfer from your computer:
user@yourlaptop:~> sftp hpc-login sftp> cd /scratch/user sftp> put myfile
Note that running the command multiple times will always overwrite the file on destination.
Advanced: High Performance Transfers with bbcp
You may also use a specialised tool called bbcp for best transfer rates. It is available on Atos HPCF. This tool is not as flexible as rsync when it comes to updating existing or partial copies, but it should be quicker when doing a one-off transfer.
For example, if you wanted to transfer a directory called mydataset in your SCRATCH in Reading, you could initiate the transfer from ECGATE:
user@yourlaptop:~> bbcp -rp -s 10 -P 2 mydataset hpc-login:/scratch/user/
The options above would perform a recursive copy, preserving file permissions and using 10 parallel streams. It would also report progress every 2 seconds.
You may find all the details in the bbcp official documentation.
Copying softlinks
bbcp will not copy soft links, so if you are copying an entire directory structure that contains them, you may need to copy them over at a second stage with another tool like rsync.