ecFlow's documentation is now on readthedocs!

Please go to https://ecflow.readthedocs.io/en/latest/index.html

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 137 Next »

Run ecflow_ui at home and interact with ecFlow suites as if you were at the Centre!

Setting up SSH access

As a prerequisite, you will need SSH access to the Centre. ECMWF Teleport gateway offers relatively hassle-free SSH access service. Follow instructions on the Centre's User Documentation pages to install and set up Teleport client on your laptop.

Each time you connect to the Centre via ssh, you will need a valid SSH certificate. Fresh certificate can be obtained from the Centre's Teleport service by executing "tsh login" command on the laptop. These certificates are valid for 12 hours. "tsh login" may ask you for your ECMWF login credentials (security token). If you already have a valid certificate, this authentication prompt will be skipped.

Connecting to ecFlow servers at the Centre

Once you have SSH access to the Centre, you can use it to connect ecflow_ui running on your laptop with ecFlow servers at ECMWF. There are two ways of doing it. Both methods rely on the Port Forwarding functionality built into OpenSSH software suite.

Method #1: Local Port Forwarding

You will start an SSH local port forwarding session on your laptop. The session will tunnel network traffic between your ecflow_ui and the Centre's servers:

  • ecflow_ui ↔ ECMWF ecFlow servers
  • ecflow_ui ↔ ECMWF logservers (in practice only 1 server possible)

Start SSH port forwarding session

Here I wish to connect to ECMWF workstation hostname machine1 where I have a ecflow server running on port 4141, also wish to connect to host machine2 where I have a ecflow server running on port 3142 (replace "user1" with your ECMWF username and "machine1", "machine2" with real ECMWF hostnames). To start the SSH tunnel on your laptop, execute:

ssh -J user1@shell.ecmwf.int user1@machine1 -C -N -L 4141:machine1:4141 -L 3142:machine2:3142

The ssh session will keep running in the terminal, listening on local ports 4141 and 3142 and printing various log messages as it forwards your ecflow network traffic to the Centre.

To access HPC job output via the log server, you need to create a tunnel access for this as well. Here we use 'logserver' as an example (replace this with qn actual logserver).

ssh -J user1@shell.ecmwf.int user1@machine1 -C -N -L 4141:machine1:4141 -L 3142:machine2:3142 -L 9316:logserver:9316

Configure your ecflow_ui

Your local ecflow_ui cannot directly access ecflow servers running at the Centre. Instead, you will connect ecflow_ui to your local SSH port forwarding session. Open ecflow_ui connection settings; make sure to use "localhost" in the "Host" fields, not the actual ECMWF hostnames. The local SSH session will securely forward your connections to the remote SSH server running inside ECMWF on machine1 and from there to the Centre's ecFlow servers.

 Redirect logserver requests to localhost

Your local ecflow_ui cannot directly access logservers running at the Centre. In Section 2.1.1 we have created a tunnel for logserver traffic. In order to send the the log requests from the local ecflow_ui to the local tunnel entrance, you need to add "127.0.0.1 logserver" alias in /etc/hosts on your laptop (replace "logserver" with an actual ECMWF logserver name).

In practice, using Local Port Forwarding method we can only connect to a single ECMWF logserver. This is because:

  • different logservers at the Centre use the same port number (9316).
  • we cannot map all of them to a single 127.0.0.1:9316 local endpoint
  • we could map each logserver:9316 to a different local port number, but we cannot reconfigure ecflow_ui to use these local port numbers (i.e. we cannot change ECF_LOGPORT=9316 defined on the ecFlow server without affecting other users).

Method #2: Dynamic Port Forwarding

SSH also offers Dynamic Port Forwarding, a.k.a. SOCKS proxy functionality. This method has some advantages over Local Port Forwarding:

  • There is no need to manually specify port number mapping for each ecflow server
  • You can use original host names in the ecFlow servers connection settigs, instead of 'localhost'
  • You are not limited to a single logserver; there is also no need to modify /etc/hosts file.

The SOCKS proxy can tunnel all network traffic from your ecflow_ui to the servers running inside the Centre:

  • ecflow_ui ↔ ECMWF ecFlow servers
  • ecflow_ui ↔ ECMWF ecFlow logservers
  • ecflow_ui ↔ ECMWF DNS servers

However, to connect with SOCKS proxy, the client application (ecflow_ui) must be able to speak SOCKS protocol. We will show how to enable it for ecflow_ui.

NOTE: if you were using Local Port Forwarding method, you will now need to:

  • remove "127.0.0.1 logserver" from your /etc/hosts
  • use actual ECMWF hostnames in the "Host" fields of ecflow_ui connection settings, instead of "localhost".

Start SOCKS proxy session

In a terminal, start the proxy session with:

% ssh -v -C -N -D 9050 -J myecuser@shell.ecmwf.int myecuser@myecworkstation

(replace "myecuser" and "myecworkstation" with your real ECMWF username and workstation name).

The session will keep running in the terminal, listening on local port 9050 and printing various log messages as it forwards your network traffic to the Centre.

SOCKS-ify your ecflow_ui

Applications which want to use SOCKS proxy must speak SOCKS protocol. Some applications can be SOCKS-ified using a tool called proxychains. The tool intercepts the application's network traffic, adds SOCKS protocol layer and redirects traffic to the proxy. Luckily, the ecflow_ui.x executable can be SOCKS-ified this way.

On MacOS, you can install proxychains tool with "brew install proxychains-ng". Many Linux distributions come with proxychains tool preinstalled.

Now comes the key part: edit the ecflow_ui lauch script and replace the "$exe" with proxychains4 "$exe" at the end of the script. On my machine, ecflow_ui launch script is installed as /opt/miniconda3/bin/ecflow_ui.

You should now be able to start ecflow_ui and interact with ecFlow suites running at the Centre.

By default, proxychains tool sends network traffic to localhost:9050. If your SOCKS proxy is listening on a different port, adjust /usr/local/etc/proxychains.conf configuration file (MacOS) accordingly.

Accessing restricted ecFlow servers

Servers with username-based access control

Many ecFlow servers at the Centre use a whitelist to only allow authorized users in. This creates a problem when connecting remotely - typically your username on the laptop will be different from your authorized ECMWF username.

There are two possible ways to access these servers:

  • you can ask the ecFlow server administrator to add your laptop username to the server's whitelist, or
  • you can create a new user account on your laptop, setting username to match the ECMWF username.

If you create new user account for ecflow_ui but want to keep using your regular account, here is a handy script. It uses Dynamic Port Forwarding method, but can be easily adapted to use Local Port Forwarding.

A script for starting SSH tunnel + ecflow_ui on a laptop as another user 
#!/bin/bash
set -e
# --------------------------------------------------------------
# A script for starting SSH SOCKS proxy and ecFlow UI as another user.
# Prerequisites:
#   * User named "myecuser" must exist on the laptop
#   * "myecuser" has set up their Teleport client to access ECMWF.
# --------------------------------------------------------------

ECMWF_USER=myecuser              # your ECMWF username
ECMWF_HOST=myecworkstation       # your ECMWF workstation name

xhost + || :
sudo -i -u "$ECMWF_USER" -- sh << SUDO

    set -e

    # Starting ssh-agent for $ECMWF_USER.
    # Teleport service requires ssh-agent running.

    ssh-agent -- sh << SSH_AGENT

        set -e

        # "tsh login" will fetch SSH certificate from
        # shell.ecmwf.int and load it onto the ssh-agent.

        tsh login -d

        # "ssh -f" will start SSH tunnel in the background.
        # The trap will terminate the tunnel on exit.

        trap 'pkill -f ssh.*-f' 0 1 2 3 15
        ssh -f -N -v -C -D9050 -J "$ECMWF_USER@shell.ecmwf.int" "$ECMWF_USER@$ECMWF_HOST"

        # finally, start the local ecFlow UI
        /Volumes/Macintosh\ HD/opt/miniconda3/bin/ecflow_ui
SSH_AGENT
SUDO

Servers with password-based access control

As an alternative to username-based access control, ecFlow offers password-based access control. See: Security(custom user).

The password-based access control must be enabled on the ecFlow server side. Discuss with your server administrator to set it up. 

When the server-side setup is done by the administrator, configure the connection settings in your ecflow_ui: 

Replace "host1" with a real ECMWF ecFlow server hostname and "myecflowuser" with your authorised ecFlow username, for which you have the password.

Known issues

  • Sometimes ecflow_ui fails to fetch the job output, job script and .sub files from the logserver. Refreshing the tab usually fixes the issue. The initial error looks like: "Output directory: [1] Failed to fetch from logserver@9316 error: The remote host closed the connection. [2] No access to path on disk!"
  • For some suites the ecflow_ui cannot retrieve job output, job script and .sub files no matter what. It happens when the log server at the centre doesn't have access to these files. ecflow_uis running at the centre have direct access to the files on disk so no problem, but remote ecflow_ui cannot access them.
  • The timeline and server load functionality currently doesn't work
  • No labels