The ectrans command allows you to transfer files securely between ECMWF and remote sites. Like the UNIX "rcp" command, ectrans requires no password to be specified for the remote host: the ECaccess gateway performs the security checking. Unlike standard FTP, ectrans is suitable for unattended file transfers in scripts, cron jobs, etc., as it avoids the problems inherent in storing passwords in text files and sending passwords across networks. Even if you don't have a local gateway installed, you can benefit from the ectrans command by using the ECMWF ECaccess gateway. Please note that in this case the transfer is not as secured as when a Member State ECaccess gateway is used.
| Table of Contents |
|---|
...
ECtrans Association
Users who wish to transfer files between ECMWF and Member State servers using ectrans need to declare one or more ectrans associations for the storage/retrieval of the remote file. This can be done either through the Web Toolkit command ecaccess-association-put (see Web Toolkit - The full featured client) or through the ECaccess Web interface of the target gateway (see The Web server). For every association (previously known as "msuser"), the hostname, login username and password, target path and transfer protocol need to be specified. Target directories can be located on:
- Member State servers running a standard FTP/SFTP service accessible from the ECaccess gateway. This is known as a "genericFtp"/"genericSftp" protocol and is the most convenient way of getting the files to the system you want, under the specified user ID.
- The server running the ECaccess gateway. This is known as a "genericFile" protocol. All users will share in a common directory the files transferred using this protocol.
- Member State servers running a proprietary application. The administrator provides ectrans with the implementation of the access protocol. The administrator can also use more complex rules to define special target locations for ECMWF users, Member State users or groups of Member State users.
Additional parameters, related to the chosen transfer protocol can be specified in the ECtrans Association.
The protocols available are the following:
- genericFtp: standard FTP protocol (ftp options).
- genericSftp: standard SFTP protocol (sftp options).
- genericFile: transmit a file on the target Gateway file system.
- genericExec: transmit and execute a file on the target Gateway file system (exec options).
The "genericFile" and "genericExec" protocols are only available on remote Gateways.
The various options available for each protocol are described in the following tables (please note the "genericFile" protocol has no option).
The ftp options
| Name | Default | Comment |
|---|---|---|
| ftp.passive | "false" | Allow selecting the Passive or Active FTP mode. In Passive mode, the FTP client initiates both connections to the server, solving the problem of firewalls filtering the incoming data port connection to the client from the server. In Active mode, the FTP client connects from a random unprivileged port (N > 1023) to the FTP server's command PORT, usually port 21. Then, the client starts listening to port N+1 and sends the FTP command |
| ftp.extended | "false" | Its use is required when connecting to a remote IPv6 server (EPSV/EPRT commands are issued instead of PASV/PORT commands when opening data channels). |
| ftp.lowPort | "false" | Allow forcing the FTP client to bind a privileged port (500 <= N <= 1023) instead of an unprivileged port (N > 1023) when using the Active FTP mode. |
| ftp.commTimeOut | "60000" (ms) | Enable/disable a timeout when opening the control channel. The value must be > 0. A timeout of zero is interpreted as an infinite timeout. |
| ftp.dataTimeOut | "60000" (ms) | Enable/disable a timeout when reading from the data channel. The value must be > 0. A timeout of zero is interpreted as an infinite timeout. |
| ftp.portTimeOut | "60000" (ms) | Enable/disable a timeout when opening the data channel specified by a PORT command. The value must be > 0. A timeout of zero is interpreted as an infinite timeout. |
| ftp.listenAddress | [none] (string) | Allow specifying the listen address used by the FTP client (e.g. when waiting for an incoming data connection). By default, the system will pick up a valid local address. A value of "0.0.0.0" will cause the binding of all the valid network interfaces. |
| ftp.sendBuffSize | "-1" | Allow specifying a buffer size when sending data on the data channel. |
| ftp.receiveBuffSize | "-1" | Allow specifying a buffer size when receiving data from the data channel. |
| ftp.setNoop | "NOOP" | Allow specifying the "No-Operation" command. This command does not affect anything at all. It performs no action other than having the server send an OK reply. This command is used to keep connections with the FTP servers "alive" (connected) while nothing is being done. |
| ftp.port | "21" | Allow specifying the port number of the remote FTP server to connect to (used to open the control channel). |
| ftp.usecleanpath | "false" | When using the "mkdir" command this option allow forcing the FTP client to fix directory pathnames (e.g. remove multiple instances of /). |
| ftp.usenlist | "false" | Force using the NLIST command on the remote FTP server when doing a directory listing instead of the standard LIST. This options is required for some FTP servers which do not allow (or do not implement) the LIST command. |
| ftp.usetmp | "true" | Force using a temporary name when transmitting a file to the remote FTP server. Once the transmission has completed successfully, the file is renamed with its final target name. |
| ftp.mkdirs | "true" | Allow creating the directory named by the target pathname, including any necessary but non-existent parent directories. |
| ftp.prefix | [none] (string) | When using the "usetmp" option, this option allow setting the prefix to use for the temporary file name (e.g. "."). |
| ftp.suffix | [none] (string) | When using the "usetmp" option, this option allow setting the suffix to use for the temporary file name (e.g. ".tmp"). |
| ftp.postConnectCmd | [none] (string) | Allow issuing an FTP command on the remote FTP server right after the control connection is established (multiple commands can be specified using the semi columns separator). |
| ftp.preCloseCmd | [none] (string) | Allow issuing an FTP command on the remote FTP server right before closing the control connection (multiple commands can be specified using the semi columns separator). |
| ftp.preGetCmd | [none] (string) | Allow issuing an FTP command on the remote FTP server right before getting a file (multiple commands can be specified using the semi columns separator). |
| ftp.prePutCmd | [none] (string) | Allow issuing an FTP command on the remote FTP server right before putting a file (multiple commands can be specified using the semi columns separator). |
| ftp.postPutCmd | [none] (string) | Allow issuing an FTP command on the remote FTP server right after putting a file (multiple commands can be specified using the semi columns separator). The parameter "$filename" can be introduced in the commands and it gives the name of the file which was just transmitted. If the "usetmp" option is used then it will give the temporary name. |
| ftp.mkdirsCmdIndex | "0" | When using the "mkdirs" option, this index allow specifying the level where to start the directory creation. For example, if the directory is "/home/uid/test/data/out/bin", an index of 3 will make sure the FTP client does not try to create the "/home" and "/home/uid" directories. |
| ftp.preMkdirsCmd | [none] (string) | Allow issuing an FTP command on the remote FTP server right before creating a directory (multiple commands can be specified using the semi columns separator). The parameter "$dirname" can be introduced in the commands and it gives the name of the directory to create. |
| ftp.postMkdirsCmd | [none] (string) | Allow issuing an FTP command on the remote FTP server right after creating a directory (multiple commands can be specified using the semi columns separator). The parameter "$dirname" can be introduced in the commands and it gives the name of the directory just created. |
| ftp.keepAlive | "0" (ms) | Allow keeping FTP control connections alive in a pool. If the connection is unused for longer than the duration specified by this option then the connection is closed and removed from the pool. A duration of zero is interpreted as no pooling. |
| ftp.useNoop | "0" (ms) | When the "keepAlive" option is configured, then this option allow sending NOOPS commands to the remote server in order to keep the connection alive. The value of this option specify the interval between 2 sending of the NOOPS command. A value of 0 deactivate the sending of the NOOPS command. A custom NOOPS command can be specified with the "setNoop" option. |
| ftp.ignoreCheck | "true" | If set, the remote size of the file is not checked after an upload has been completed. If not set, the remote size of the uploaded file is checked and compared against the size of the source file. |
| ftp.ignoreDelete | "true" | If not set, the FTP client tries to delete the target file before the upload is processed. |
| ftp.deleteOnRename | "true" | If set, the FTP client tries to delete the target file before doing a rename (e.g. when using temporary file names with the "usetmp" option). |
| ftp.retryAfterTimeoutOnCheck | "false" | If set, the FTP client will try to reconnect if the control connection has timed-out during the data transmission. |
| ftp.keepControlConnectionAlive | "false" | If set, the FTP client will keep sending NOOPS commands through the control channel while waiting for a data transmission to complete. |
| ftp.mksuffix | "false" | If set, the FTP client will generate a random suffix for the "usetmp" option with 3 characters (e.g. ".3te"). |
| ftp.usesuffix | "false" | If set and the "mksuffix" is set as well then the suffix specified in the "suffix" option will be concatenated to the generated suffix (e.g. ".3te.tmp"). |
| ftp.cwd | [none] (string) | Allow changing directory just after the login (a "cd" command is issued with the specified directory). |
| ftp.nopassword | "false" | Allow connecting to a remote FTP server with no password required (e.g. with anonymous users). |
The sftp options
| Name | Default | Comment |
|---|---|---|
| sftp.usetmp | "true" | Force using a temporary name when transmitting a file to the remote SFTP server. Once the transmission has completed successfully, the file is renamed with its final target name. |
| sftp.mkdirs | "true" | Allow creating the directory named by the target pathname, including any necessary but non-existent parent directories. |
| sftp.prefix | [none] (string) | When using the "usetmp" option, this option allow setting the prefix to use for the temporary file name (e.g. "."). |
| sftp.suffix | [none] (string) | When using the "usetmp" option, this option allow setting the suffix to use for the temporary file name (e.g. ".tmp"). |
| sftp.chmod | [none] (integer) | If set, the file mode will be changed according to the value of this option once it has uploaded a file successfully (e.g. "640"). |
| sftp.ignoreCheck | "false" | If set, the remote size of the file is not checked after an upload has been completed. If not set, the remote size of the uploaded file is checked and compared against the size of the source file. |
| sftp.execCode | "0" | Allow specifying the expected exit code of the command (or list of commands) started with the "execCmd" option. If the exit code is different from the one specified in this parameter then the transmission is tagged as failed. |
| sftp.execCmd | [none] (string) | Allow specifying a command (or list of commands) to execute on the SSH server once a file has been transmitted (multiple commands can be specified using the semi columns separator). The parameter "$filename" can be introduced in the commands and it gives the name of the file transmitted (source or target depending if the transmission was a upload or a download). |
| sftp.kex | [none] (string) | Allow specifying the key exchange algorithms to allow (e.g. "ecdh-sha2-nistp256,ecdh-sha2-nistp384"). The list of valid algorithms can be found at http://www.jcraft.com/jsch/README (multiple algorithms can be specified using the columns separator). By default all supported algorithms are allowed. |
| sftp.cipher | [none] (string) | Allow specifying the cipher algorithms to allow (e.g. "aes128-cbc,3des-cbc,blowfish-cbc"). The list of valid algorithms can be found at http://www.jcraft.com/jsch/README (multiple algorithms can be specified using the columns separator). By default all supported algorithms are allowed. |
| sftp.compression | [none] (string) | Allow specifying the compression algorithms to allow (e.g. "zlib,none"). The list of valid algorithms can be found at http://www.jcraft.com/jsch/README (multiple algorithms can be specified using the columns separator). By default all supported algorithms are allowed. |
| sftp.fingerPrint | [none] (string) | Allow specifying a key's fingerprint. It is a lower-case hexadecimal representation of the MD5 of a key (e.g. "22:fb:ee:fe:18:cd:aa:9a:9c:78:89:9f:b4:78:75:b4"). If a key's fingerprint is defined then it is compared against the key's fingerprint of the remote SFTP server when connecting. If it does not match then the connection is rejected. |
| sftp.passPhrase | [none] (string) | The passphrase necessary to access the private key when the "privateKey" or "privateKeyFile" option is used. This option is only required if the private key is protected. |
| sftp.privateKeyFile | [none] (string) | Adds a private key to be used for public-key authentication. The private key should be in the PEM format in a file accessible from the ECaccess Gateway (the file permissions should be set accordingly). This option contains the name of this file (e.g. "/home/uid/.ssh/mykey.pem"). |
| sftp.privateKey | [none] (string) | Adds a private key to be used for public-key authentication. The private key should be in the PEM format. |
| sftp.listenAddress | [none] (string) | Allow specifying the source address used by the SFTP client when connecting to the remote SFTP server. By default, the system will pick up a valid local address. |
| sftp.sessionTimeOut | "60000" (ms) | This value is used as the socket timeout parameter, and also as the default connection timeout. The value must be > 0. A timeout of zero is interpreted as an infinite timeout. |
| sftp.connectTimeOut | [sessionTimeOut] | This value is used as the connection timeout. The value must be > 0. A timeout of zero is interpreted as an infinite timeout. |
| sftp.mksuffix | "false" | If set, the SFTP client will generate a random suffix for the "usetmp" option with 3 characters (e.g. ".3te"). |
| sftp.port | "22" | Allow specifying the port number of the remote SFTP server to connect to. |
The exec options
| Name | Default | Comment |
|---|---|---|
| Anchor | ||||
|---|---|---|---|---|
|
...