For transfers to ECMWF, we recommend using rsync which will transfer the files over an ssh connection. For that, you will need to have Teleport configured with the appropriate settings in your ssh config file.

Any file transfer tool that supports SSH and the ProxyJump feature should work, such as the command line tools sftp or scp. Alternatively, you may also use the Linux Virtual Desktop and its folder sharing capabilities to copy local files to your ECMWF's HOME or PERM.

For transfers from ECMWF to external sites, you may use tools such as ectrans, ftp, scp, sftp or rsync, depending on the protocol used on the external site.

Transfers to ECMWF

No filesystems from other platforms are cross-mounted, so you will need to copy over what you need.

First of all, make sure you have set up password-less authentication as described in HPC2020: How to connect before proceeding.

...

Note
All examples are done with the generic HPCF login node hpc-login, but you should use ecs-login if you don't have access to the full HPCF service. Please note that the examples below assume that you have a valid teleport session established.

...

Transferring a directory tree between

...

your computer and the Atos HPCF with rsync

This is the recommended and most versatile option. You may initiate the transfer from ECGATEyour computer for the standard set of filesystems:

user@ecgb11user@yourlaptop:~> rsync -avz $SCRATCH/mydataset hpc-login:/scratch/user/

or alternatively from Atos HPCF:

user@aa6-100:~> rsync -avz ecgate:/scratch/group/user/mydataset $SCRATCH/

Both solutions are equivalent, and This command can be run multiple times. Only new , since only new or modified files will be transferred.

Transferring a single file between your computer and the Atos HPCF with scp

You may initiate the transfer from your computer:

user@yourlaptop:~> scp myfile hpc-login:/scratch/user/

Note that running the command multiple times will always overwrite the file on destination.

Transferring a single file between your computer and the Atos HPCF with sftp

You may initiate the transfer from your computer:

user@yourlaptop:~> sftp hpc-login
sftp> cd /scratch/user
sftp> put myfile

Note that running the command multiple times will always overwrite the file on destination.

Transferring a single file between your computer and the Atos HPCF with rclone

rclone config file

[hpc-login]
type = sftp
ssh = ssh hpc-login

 You may initiate the transfer from your computer:

user@yourlaptop:~> rclone -P copy myfile hpc-login:/scratch/user

rclone will not copy the file if the remote already has the latest version.

Transferring files between your computer and the Atos HPCF or ECFS domains via ECaccess gateway

For a data transfer from an Internet site to ECMWF, invoke a connection to the ECaccess gateway in Bologna:

user@yourlaptop:~> sftp user@boaccess.ecmwf.int
sftp> cd ECSCRATCH
sftp> put MyFile

Fetching data from ECMWF to your computer

Transferring a directory tree between the Atos HPCF and your computer with rsync

This is the recommended and most versatile option. You may initiate the transfer from your computer for the standard set of filesystems:

:~> rsync -avz

hpc-login:

/

scratch/

or from $PERM on ecgate to $PERM on Atos HPCF running the command on the latter use:

user@yourlaptop

rsync -avz $HOME/mydataset 'hpc-login:$HOME/'

user@aa6-100:~> rsync -avz 'ecgate:${HOME/home/perm}/mydataset' $PERM/

The single quotes are needed here to ensure the $HOME variable takes its value on the remote rather than the local host.

user@leap42-workstation:~> rsync -avz -e "ssh -x -o ProxyCommand='ssh -q -x -W %h:%p ecgate'" /var/tmp/user/mydataset hpc-login:/perm/user/mydataset

Advanced: High Performance Transfers with bbcp

You may also use a specialised tool called bbcp for best transfer rates. It is available on Atos HPCF, as well as Reading Based platforms Cray HPCF and ECGATE. This tool is not as flexible as rsync when it comes to updating existing or partial copies, but it should be quicker when doing a one-off transfer.

For example, if you wanted to transfer a directory called mydataset in your SCRATCH in Reading, you could initiate the transfer from ECGATE:

user@ecgb11:~> bbcp -rp -s 10 -P 2 $SCRATCH/mydataset hpc-login:/scratch/user/

or alternatively from Atos HPCF:

user@aa6-100:~> bbcp -rp -s 10 -P 2 ecgate:/scratch/group/user/mydataset $SCRATCH/

The options above would perform a recursive copy, preserving file permissions and using 10 parallel streams. It would also report progress every 2 seconds.

You may find all the details in the bbcp official documentation.

user/mydataset .

This command can be run multiple times, since only new or modified files will be transferred.

Transferring a directory tree between the Atos HPCF and your computer with scp

You may initiate the transfer from your computer:

user@yourlaptop:~> scp hpc-login:/scratch/user/myfile .

Note that running the command multiple times will always overwrite the file on destination.

Transferring a directory tree between the Atos HPCF and your computer with sftp

You may initiate the transfer from your computer:

user@yourlaptop:~> sftp hpc-login
sftp> cd /scratch/user
sftp> get mydir

Note that running the command multiple times will always overwrite the file on destination.

Transferring a directory tree between the Atos HPCF and your computer with rclone

rclone config file

[hpc-login]
type = sftp
ssh = ssh hpc-login

 You may initiate the transfer from your computer:

user@yourlaptop:~> rclone -P copy hpc-login:/scratch/user/mydataset mydataset

rclone will not copy the file if the remote already has the latest version, and will use parallel copies of multiple files to speed up the transfers.

Transferring a directory tree between the Atos HPCF or ECFS domains and your computer via ECaccess gateway

For a data transfer from an Internet site to ECMWF, invoke a connection to the ECaccess gateway in Bologna:

user@yourlaptop:~> sftp user@boaccess.ecmwf.int
sftp> cd ECHOST/hpc-login/hpcperm/user
sftp> get MyFile

Pushing data from ECMWF to external sites

There are various methods for transferring files from ECMWF to remote sites. Which is best suited for your requirements depends on the configuration of the remote system. If you are not sure what it is, please ask your Computing Representative or system administrator.

In general you have the following options:

ectrans

The ectrans command, part of the ECaccess framework, allows files to be transferred securely and unattended, as it does not require a password to be specified for the remote host: the ECaccess gateway performs the security checking. Ideally, an ECaccess gateway should be installed at the remote site to which you want to transfer; if there is no remote site ECaccess gateway, you can use ectrans via the ECMWF gateway (ecaccess.ecmwf.int), provided that the destination site is accessible from the ECMWF gateway via (s)ftp.

Before you can make use of ectrans, you need to declare an ECtrans association (msuser) for the storage/retrieval of the remote file. This can be done either with the ECaccess tool ecaccess-association-put or through the ECaccess web interface of the gateway you want to transfer to (or at http://ecaccess.ecmwf.int , if no gateway is installed at the remote site). For every msuser declaration, the hostname and the login username and password are requested and stored on the gateway in encrypted form. After these preliminaries you should be able to use ectrans as described in its help page:

usage: ectrans [-gateway name] -remote msuser@[destination] \                                                                                                                              
          [-get|-put] -source [ec:|ectmp:]filename [args ...] (*)                                                                                                                          
        ectrans -check requestID (*)                                                                                                                                                       
                                                                                                                                                                                           
 -gateway  {arg} - access gateway name (default (**): ecaccess.ecmwf.int)
 -remote   {arg} - access method (default (**): *none*)
 -source   {arg} - source file name
 -target   {arg} - target file name (default: same as -source)
 -mailto   {arg} - target email address (default: current user)
 -lifetime {arg} - lifetime of the file in the spool (default: 1w) (***) (****)
 -delay    {arg} - transmission delay (default: immediate transfer) (***) (****)
 -at       {arg} - transmission date (default: immediate transfer) (****)
 -format   {arg} - define the date format as used with -at (default: yyyyMMddHHmmss)
 -retryCnt {arg} - define the number of retries (default: async=144, sync=0)
 -retryFrq {arg} - define the frequency of retries (default: async=10m, sync=1m) (***)
 -maxTime  {arg} - define the maximum transfer duration (default: 12h) (***)
 -priority {arg} - transmission priority 0-99 (default: 99) (****)
 -put            - interactive/synchronous transfer (no spool)
 -get            - interactive/synchronous pull (rather than push) file
 -onsuccess      - mail sent on successful transfer
 -onfailure      - mail sent when transfer has failed
 -onretry        - mail sent when transfer is retried
 -keep           - keep the request in the spool till expiration (****) (*****)
 -remove         - always remove the request from the spool (****) (*****)
 -reject         - if existing target file (default)
 -append         - if existing target file
 -resume         - if existing target file
 -overwrite      - if existing target file
 -verbose        - verbose mode on
 -version        - print version number
 -help           - this message

    (*) If successful, a requestID is returned, which can be used in
          check requests. Exit code is 0 on success and >0 otherwise.
   (**) The default values depend on the GATEWAY or REMOTE environment
          variables.
  (***) Duration in weeks, days, hours, minutes or seconds (e.g. 1w|2d).
 (****) These options are only relevant when the spool is used. The spool
          is no used during interactive transfers (-get and -put options).
(*****) By default, successful requests are removed from the spool and
          failed requests are kept in the spool till expiration.

Ectrans can be regarded as an extended ftp, which offers e.g. direct access to ECFS files, restart facility for failed transfers etc. For more details please refer to the ECaccess documentation or reach us via the ECMWF Support Portal.

rsync/sftp/scp

The most popular SSH-based transfer commands such as rsync, sftp and scp are fully supported as part of SSH to directly connect to remote sites via the Internet. No proxies are needed, but the only supported remote ports are 22, the standard for SSH and non-standard 2222.

rclone

You may also use rclone to push files to remote destinations supporting a wide range of protocols.  You can make rclone available in your session with:

module load rclone

Note that mounting of remote filesystems is not available.

ftp and lftp

FTP (File Transfer Protocol) can be used without a proxy to connect to remote servers on the standard port, 21, and non-standard 2121. You may use the ftp classic client, or the lftp loading the appropriate module:

module load lftp
lftp yourftpserver

When transferring data, make sure you are using the right transfer type (binary/ASCII). Attempting to transfer a binary file as ASCII will result in a corrupted binary file

...

...

.