For transfers to ECMWF, we recommend using rsync which will transfer the files over an ssh connection. For that, you will need to have Teleport configured with the appropriate settings in your ssh config file.
Any file transfer tool that supports SSH and the ProxyJump feature should work, such as the command line tools sftp or scp. Alternatively, you may also use the Linux Virtual Desktop and its folder sharing capabilities to copy local files to your ECMWF's HOME or PERM.
...
Note that running the command multiple times will always overwrite the file on destination.
Transferring
...
a single file between your computer and the Atos HPCF
...
with rclone
Tip | ||||
---|---|---|---|---|
| ||||
You will need to have rclone installed and configured in your computer. Check the location of your rclone configuration file with:
and add the following snippet to configure a new remote:
|
You
For a data transfer from an Internet site to ECMWF, invoke a connection to the ECaccess gateway in Bologna:
No Format |
---|
user@yourlaptop:~> sftp user@boaccess.ecmwf.int
sftp> cd ECSCRATCH
sftp> put MyFile |
Fetching data from ECMWF to your computer
Transferring a directory tree between the Atos HPCF and your computer with rsync
This is the recommended and most versatile option. You may initiate the transfer from your computer for the standard set of filesystems:
No Format |
---|
user@yourlaptop:~> rsyncrclone -avzP copy myfile hpc-login:/scratch/user/mydataset . |
This command can be run multiple times, since only new or modified files will be transferred.
Tip |
---|
You may add the |
Transferring a directory tree between the Atos HPCF and your computer with scp
|
rclone will not copy the file if the remote already has the latest version.
Transferring files between your computer and the Atos HPCF or ECFS domains via ECaccess gateway
For a data transfer from an Internet site to ECMWF, invoke a connection to the ECaccess gateway in BolognaYou may initiate the transfer from your computer:
No Format |
---|
user@yourlaptop:~> scp hpc-login:/scratch/user/myfile . |
...
sftp user@boaccess.ecmwf.int
sftp> cd ECSCRATCH
sftp> put MyFile |
Fetching data from ECMWF to your computer
Transferring a directory tree between the Atos HPCF and your computer with
...
rsync
This is the recommended and most versatile option. You may initiate the transfer from your computer for the standard set of filesystems:
No Format |
---|
user@yourlaptop:~> rsync -avz hpc-login:/scratch/user/mydataset . |
This command can be run multiple times, since only new or modified files will be transferred.
Tip |
---|
You may add the |
Transferring a directory tree between the Atos HPCF and your computer with scp
You may initiate the transfer from your computer:
No Format |
---|
user@yourlaptop:~> scp hpc-login:/scratch/user/myfile . |
Note that running the command multiple times will always overwrite the file on destination.
Transferring a directory tree between the Atos HPCF and your computer with sftp
You may initiate the transfer from your computer:
No Format |
---|
user@yourlaptop:~> sftp hpc-login
sftp> cd /scratch/user
sftp> get mydir |
Note that running the command multiple times will always overwrite the file on destination.
Transferring a directory tree between the Atos HPCF and your computer with rclone
Tip | ||||
---|---|---|---|---|
| ||||
You will need to have rclone installed and configured in your computer. Check the location of your rclone configuration file with:
and add the following snippet to configure a new remote:
|
You You may initiate the transfer from your computer:
No Format |
---|
user@yourlaptop:~> rclone -P sftpcopy hpc-login sftp> cd :/scratch/user sftp> get myfile/mydataset mydataset |
rclone will not copy the file if the remote already has the latest version, and will use parallel copies of multiple files to speed up the transfersNote that running the command multiple times will always overwrite the file on destination.
Transferring a directory tree between the Atos HPCF or ECFS domains and your computer via ECaccess gateway
...
The most popular SSH-based transfer commands such as rsync, sftp and scp are fully supported as part of SSH to directly connect to remote sites via the Internet. No proxies are needed, but the only supported remote ports are 22, the standard for SSH and non-standard 2222.
rclone
You may also use rclone to push files to remote destinations supporting a wide range of protocols. You can make rclone available in your session with:
No Format |
---|
module load rclone |
Note that mounting of remote filesystems is not available.
ftp and lftp
FTP (File Transfer Protocol) can be used without a proxy to connect to remote servers on the standard port, 21, and non-standard 2121. You may use the ftp classic client, or the lftp loading the appropriate module:
...