Unattended file transfer - ectrans

Unattended file transfer - ectrans

The ectrans command allows you to transfer files securely between ECMWF and remote sites. Like the UNIX "rcp" command, ectrans requires no password to be specified for the remote host: the ECaccess gateway performs the security checking. Unlike standard FTP, ectrans is suitable for unattended file transfers in scripts, cron jobs, etc., as it avoids the problems inherent in storing passwords in text files and sending passwords across networks. Even if you don't have a local gateway installed, you can benefit from the ectrans command by using the ECMWF ECaccess gateway. Please note that in this case the transfer is not as secured as when a Member State ECaccess gateway is used.

4.1  Target location

Users who wish to transfer files between ECMWF and Member State servers using ectrans need to declare one or more ectrans associations for the storage/retrieval of the remote file. This can be done either through the Web Toolkit command ecaccess-association-put (see section 5.8) or through the ECaccess Web interface of the target gateway (see section 7.4). For every association (previously known as "msuser"), the hostname, login username and password, target path and transfer protocol need to be specified. Target directories can be located on:

• Member State servers running a standard FTP/SFTP service accessible from the ECaccess gateway. This is known as a "genericFtp"/"genericSftp" protocol and is the most convenient way of getting the files to the system you want, under the specified user ID.
• The server running the ECaccess gateway. This is known as a "genericFile" protocol. All users will share in a common directory the files transferred using this protocol.
• Member State servers running a proprietary application. The administrator provides ectrans with the implementation of the access protocol. The administrator can also use more complex rules to define special target locations for ECMWF users, Member State users or groups of Member State users.

4.2   ectrans command

With the ectrans command, Member State users can initiate secure file transfers between ECMWF (ecgate or HPCF systems) and a Member State server, as defined in the ectrans association. When ectrans is used to put a file (from ECMWF to a remote server), the ECaccess Server will spool the file in the user's "ectrans" transfer queue. If the connection between the ECMWF and the remote gateway is down or if any error occurs, the file will be kept in the spool area at ECMWF and you can resume the transfer either through the web interface or with the Web Toolkit command ecaccess-ectrans-restart. When ectrans is used to get a file (from a remote server to ECMWF) the transfer will fail by default, if the connection between the ECMWF and the remote gateway is down. A retry mechanism is available for all types of transfers. To show the ectrans usage:

> ectrans -help
usage: ectrans [-gateway name] -remote association@[protocol] \
          [-get|-put] -source [ec:|ectmp:]filename [args ...] (*)
        ectrans -check requestID (*)

 -gateway  {arg} - access gateway name (default (**): ecaccess.ecmwf.int)
 -remote   {arg} - association and (optional) protocol (default (**): *none*)
 -source   {arg} - source file name
 -target   {arg} - target file name (default: same as -source)
 -mailto   {arg} - target email address (default: current user)
 -lifetime {arg} - lifetime of the file in the spool (default: 1w) (***) (****)
 -delay    {arg} - transmission delay (default: immediate transfer) (***) (****)
 -at       {arg} - transmission date (default: immediate transfer) (****)
 -format   {arg} - define the date format as used with -at (default: yyyyMMddHHmmss)
 -retryCnt {arg} - define the number of retries (default: async=144, sync=0)
 -retryFrq {arg} - define the frequency of retries (default: async=10m, sync=1m) (***)
 -maxTime  {arg} - define the maximum transfer duration (default: 12h) (***)
 -priority {arg} - transmission priority 0-99 (default: 99) (****)
 -put            - interactive/synchronous transfer (no spool)
 -get            - interactive/synchronous pull (rather than push) file
 -onsuccess      - mail sent on successful transfer
 -onfailure      - mail sent when transfer has failed
 -onretry        - mail sent when transfer is retried
 -keep           - keep the request in the spool till expiration (****) (*****)
 -remove         - always remove the request from the spool (****) (*****)
 -reject         - if existing target file (default)
 -append         - if existing target file
 -resume         - if existing target file
 -overwrite      - if existing target file
 -verbose        - verbose mode on
 -version        - print version number
 -help           - this message

    (*) If successful, a requestID is returned, which can be used in
          check requests. Exit code is 0 on success and >0 otherwise.
   (**) The default values depend on the GATEWAY or REMOTE environment
          variables.
  (***) Duration in weeks, days, hours, minutes or seconds (e.g. 1w|2d).
 (****) These options are only relevant when the spool is used. The spool
          is no used during interactive transfers (-get and -put options).
(*****) By default, successful requests are removed from the spool and
          failed requests are kept in the spool till expiration.

The "reject", "append", "resume" and "overwrite" options are mutually exclusive and determine what to do if there is an existing target file. The "mailto" option specifies an email address to be notified in case of a successful (option "onsuccess") and/or a failed transfer (option "onfailure"). The "check" option prints the status of the specified request on the standard output. The transfer status, which can be checked with the ecaccess-ectrans-list command or the Web interface, can takes values as listed in table 1. Table 1: Transfer status.

4.2.1  Transfer to a Member State host via gateway

To transfer file "fff" from the current working directory on "ecgate" to the "genericFtp" protocol of the use "myUser" on the ECaccess gateway "ecaccess.meteo.ms":

> ectrans -gateway ecaccess.meteo.ms \
          -remote myUser@genericFtp
          -source fff \
          -verbose
verbose: gateway=ecaccess.meteo.ms
verbose: echost=ecgate.ecmwf.int
verbose: ecport=644
verbose: action=spool
verbose: ecuser=uid
verbose: source=fff
verbose: target=fff
verbose: keep=false
verbose: remove=false
verbose: option=reject
verbose: lifetime=1w
verbose: delay=(none)
verbose: at=(now)
verbose: format=yyyyMMddHHmmss
verbose: retryCnt=144
verbose: retryFrq=10m
File to upload (5140480 bytes)
9442903031

When a request has been spooled successfully, a requestID is returned immediately. ectrans will then return the exit code 0. The requestID can be used to reference the transfer, using the interface described in section 8 or with the command ecaccess-ectrans-list. If the file is not successfully spooled, an error message is printed and the ectrans return code is -1.

4.2.2  Transfer from a Member State host via gateway

To transfer file "fff" at the "genericFtp" protocol of the "myUser" association of the ECaccess gateway "ecaccess.meteo.ms" to the current directory at ECMWF:

> ectrans gateway ecaccess.meteo.ms \
          -remote myUser@genericFtp \
          -get -source fff \
          -verbose
gateway: ecaccess.ecmwf.int
echost: ecgate.ecmwf.int
ecport: 644
action: get
ecuser: uid
target: fff
source: fff
keep  : false
option: reject
File to download (0 bytes)
5140480 bytes to download

When the request has been carried out successfully, the result is returned immediately. Transfers from a Members State to ECMWF are not spooled; they are carried out synchronously. The ectrans return code is 0 if the file has been transferred successfully or -1 if the file has not been transferred successfully.