You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 185 Next »

Teleport is the replacement for the ECaccess SSH service. Both services are currently available and can be widely used at this point.

However, once ECMWF's systems are fully operational in the Bologna Data Centre and the Reading-based hosts (the Cray HPCF, ECGATE and others) have been decommissioned, Teleport SSH will be the only way to connect using ssh to ECMWF's Bologna-based hosts (the Atos HPCF and ECGATE Services).  The ECaccess service will not support interactive ssh connections to the Bologna-based hosts. 

Teleport is software which provides an SSH Jump Host (or Bastion host) service in a secure, modern way, with support for role-based access control and single sign-on. At the moment there are two gateways available:

  • shell.ecmwf.int based in Reading Datacentre, best to access Reading-based hosts (Cray HPCF, ECGATE and others).
  • jump.ecmwf.int based in Bologna Datacentre, best to access Bologna-based hosts (Atos HPCF)

The Teleport gateway SSH Host Keys are currently: 

shell.ecmwf.int SHA256:ST5P3QlRZdI88o79ozjPdp0+FWTczckLTKzGD2z3xmU

jump.ecmwf.int  SHA256:yAJzmK5nCp9R0TIxkfdaLjJt3uf3S0QkxjGX18Ws5EM

Please report any feedback or issues through the ECMWF Support Portal .

Overview

The Teleport service provides:

  • Single SSH hop from client systems anywhere on the internet to servers inside ECMWF (ecGate, HPC, etc)
  • Re-authentication required only every 12 hours (once per day)
  • Integration with standard tools such as the OpenSSH ssh client, scp, and ssh-agent
  • Web-SSH interface for in-browser terminal access, with scp
  • X11 and Port forwarding

The single sign-on step is performed using an application called "tsh", every 12 hours.

After that you use standard ssh or scp to connect to systems inside ECMWF.

Alternatively you can have simple terminal access in a web browser. 

Downloading tsh 

The tsh application is required to perform user authentication once every 12 hours.

MacOS users can use Homebrew for installation: brew install teleport.

Alternatively, and for Windows, download from the Gravitational website and place into your $PATH.

The binary is available for Linux 32-bit, 64-bit, and ARM, as well as Windows 64-bit and a signed package for MacOS.

User Authentication

Once every 12 hours, you will need to refresh your tokens with the tsh command, through your web browser.

SSH connections can remain active for longer than 12 hours, but new ones will require re-authentication.

First Time

Run tsh, giving the location of our gateways. For platforms based in Reading:

tsh login --proxy=shell.ecmwf.int:443

or for Bologna hosts such as the Atos HPCF:

tsh login --proxy=jump.ecmwf.int:443

Your default web browser will open and you should login with your email address, ECMWF password, and then HID (ActivID) Token code.

If you're already logged in to the ECMWF website, or have recently logged in to this service, the password prompt might be skipped.

Subsequent Occasions

if you only use one of the gateways, on subsequent occasions you may just do:

tsh login

If you use both, you will need to specify which one you want to login, otherwise it will just pick the last one used.

Using both gateways simultaneously

There is currently a limitation in the SSH agent that can only use one set of keys (the one for the last tsh login). If you wish to use both on the same session, you would need to tell ssh what keys need to be used explicitly. One way to do it is to define the following entry in your ~/.ssh/config (replace <email_address> by the actual value. 

# Teleport gateways
# Temporary workaround to allow concurrent usage of both gateways
Host shell.ecmwf.int jump.ecmwf.int
    IdentityFile ~/.tsh/keys/%h/<email_address>
    CertificateFile ~/.tsh/keys/%h/<email_address>-ssh/%h-cert.pub
    HostKeyAlgorithms ssh-rsa-cert-v01@openssh.com
    PubkeyAcceptedAlgorithms +ssh-rsa-cert-v01@openssh.com

If you are not sure what email address you need to use, just run the tsh login once and run the following:

ls ~/.tsh/keys/*/*.pub

Connecting to hosts through the gateway

Windows users should skip to our Guide for Windows SSH to ECMWF.

OpenSSH 7.3 or later has a simple command line option to connect via our gateway to the destination-host:

ssh -J username@shell.ecmwf.int username@destination-host
ssh -J username@jump.ecmwf.int username@destination-host

For example, if your username is ab0 and you wish to connect to ecgate:

ssh -J ab0@shell.ecmwf.int ab0@ecgate

And if you wish to connect to Atos HPCF:

ssh -J ab0@jump.ecmwf.int ab0@hpc-login
# For users without HPC access:
ssh -J ab0@jump.ecmwf.int ab0@ecs-login


The OpenSSH configuration setting for this is named ProxyJump, e.g. add the following lines in the ~/.ssh/config file on you client system:

# For ecgate and Cray HPCF
Host ecg* cc* 
  User ab0
  ProxyJump ab0@shell.ecmwf.int

# For Atos HPCF
Host a?-* a??-* hpc-* hpc2020-* ecs-*
  User ab0
  ProxyJump ab0@jump.ecmwf.int

See the Legacy Configuration note below if your ssh client is older than 7.3.

If your connection fails after working for some time, it could be because your tokens have expired. You can check them:

$ tsh status
> Profile URL:  https://shell.ecmwf.int:443
  Logged in as: firstname.lastname@ecmwf.int
  Cluster:      shell.ecmwf.int
  Roles:        *
  Logins:       ab0
  Valid until:  2020-06-22 23:26:30 +0100 BST [EXPIRED]
  Extensions:   permit-X11-forwarding, permit-agent-forwarding, permit-port-forwarding, permit-pty

From OpenSSH 8.4 the client may refuse to connect with the cryptic message: "Connection closed by UNKNOWN port 65535".

This is because the Teleport system has to remain compatible with some old OpenSSH server versions. The fix is to add this extra lines to your OpenSSH configuration:

Host shell.ecmwf.int jump.ecmwf.int
    HostKeyAlgorithms ssh-rsa-cert-v01@openssh.com
    PubkeyAcceptedAlgorithms +ssh-rsa-cert-v01@openssh.com

Destination Hosts available

The hosts directly available through the Reading Teleport gateway (shell.ecmwf.int) are:

  • ECGATE (interactive node only)
  • CCA and CCB login nodes

For hosts based in Bologna through the Bolgona Teleport gateway (jump.ecmwf.int):

  • Atos HPCF, including ECS

Configuring password-less login

This configuration enables single-hop ssh (using ProxyJump) to other ECMWF hosts.

Not required for ECGATE, CCA/CCB or Atos HPCF login nodes, Linux physical workstations and Linux VDI.

Add the Teleport certificate authority to your ~/.ssh/authorized_keys file, on the relevant system at ECMWF, e.g. ecgate, cca:

curl -fs https://nexus.ecmwf.int/repository/internal-teleport-configs/prod/teleport_user_ca.pub >> ~/.ssh/authorized_keys

On cca/ccb, you will need to load the curl module beforehand.

This configuration will allow access to any host which mounts the same $HOME directory.

Terminal Access in a Web Browser

You can open a tabbed terminal in the web browser, with support for SCP upload and download.

Only available for ECGATE and CCA/CCB login nodes.

Browse to  http://webshell.ecmwf.int/destination-host/username (replacing destination-host and username with your selection).

For example, if your username is ab0 and you wish to connect to ecgate:

http://webshell.ecmwf.int/ecgate/ab0

The destination host should be just the short name, without "ecmwf.int".

If you open another tab inside the web terminal, use the QUICK LAUNCH box and enter "username@destination-host:22", for example ab0@ecgate:22.

The web terminal works very well to access tmux or screen running on the destination host.

Windows Clients

There are various ways to initiate SSH from Windows 10, so it depends on your system and your preferences.

We recommend using the Windows Subsystem for Linux if you can (on your own machine), followed by starting the SSH Agent and then connecting as for Linux/MacOS systems.

Alternatively, you can use:

  1. Install MobaXterm if it is not already on your system
  2. Download tsh (you may need to instruct antivirus software to ignore the file)
  3. Start MobaXterm
  4. Login using tsh (you will always need to specify the --proxy setting)
  5. Use the following in $HOME/.ssh/config (MobaXterm's home):
Host shell.ecmwf.int jump.ecmwf.int
    UserKnownHostsFile /drives/c/Users/<windows_user>/.tsh/known_hosts
    IdentityFile /drives/c/Users/<windows_user>/.tsh/keys/%h/<email_address>
    CertificateFile /drives/c/Users/<windows_user>/.tsh/keys/%h/<email_address>-ssh/%h-cert.pub

# For ecgate and Cray HPCF
Host ecg* cc*
    User ab0
    HostKeyAlgorithms +ssh-rsa*
    PubkeyAcceptedKeyTypes +ssh-rsa*
    ProxyJump ab0@shell.ecmwf.int
    IdentityFile /drives/c/Users/<windows_user>/.tsh/keys/shell.ecmwf.int/<email_address>
    CertificateFile /drives/c/Users/<windows_user>/.tsh/keys/shell.ecmwf.int/<email_address>-ssh/shell.ecmwf.int-cert.pub
  
# For Atos HPCF
Host a?-* a??-* hpc-* hpc2020-* ecs-*
    User ab0
    ProxyJump ab0@jump.ecmwf.int
    IdentityFile /drives/c/Users/<windows_user>/.tsh/keys/jump.ecmwf.int/<email_address>
    CertificateFile /drives/c/Users/<windows_user>/.tsh/keys/jump.ecmwf.int/<email_address>-ssh/jump.ecmwf.int-cert.pub
  1. Start the Windows SSH Agent Service
  2. Download tsh (you may need to instruct antivirus software to ignore the file)
  3. Login using tsh (you will always need to specify the --proxy setting)
  4. Use an SSH config as below:
# Windows currently has a bug, you need the full path to ssh or you will get:
# posix_spawn: No such file or directory

Host *.ecmwf.int
  User ab0
  ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -q -W %h:%p ab0@shell.ecmwf.int

# this assumes the SSH Agent is running, otherwise add:
# Host shell.ecmwf.int
#   IdentityFile ~/.tsh/keys/shell.ecmwf.int/firstname.lastname@ecmwf.int
#   CertificateFile ~/.tsh/keys/shell.ecmwf.int/firstname.lastname@ecmwf.int-ssh/shell.ecmwf.int-cert.pub

The Windows 10 Terminal is a decent tabbed command line interface (albeit with no X11 support).

{
  "guid": "{717406b0-06cb-454c-a0c4-875267fa373d}", # run "[guid]::NewGuid()" in a PowerShell to generate this
  "name": "ecGate",
  "commandline": "ssh ab0@ecgate.ecmwf.int"
  "suppressApplicationTitle": true,
  "hidden": false,
  "fontSize": 10,
  "fontFace": "Fira Code",
  "cursorShape": "filledBox",
  "cursorColor": "#073642",
  "colorScheme": "Solarized Dark",
},

Other Notes

SSH Agent is required

If you have logged in but ssh fails to connect, it may be that your SSH agent is not running.

The Agent can be started and tokens refreshed this way:

eval $(ssh-agent -s)
tsh logout
tsh login

And this will make sure the Agent continues to run in your environment:

echo 'eval $(ssh-agent -s)' >> ~/.bash_profile

SCP, X11, Agent, and Port Forwarding

SCP, Agent forwarding, X11 forwarding, and Port forwarding (including SOCKS proxy), all work through the Teleport gateway. A nice application of port forwarding with Teleport is the use of the ecflow_ui client on your local system to follow ecflow suites running at ECMWF.  See Teleport - using local ecflow_ui for more details.

For scp you can use the -o option:

scp -o ProxyJump=ab0@shell.ecmwf.int ab0@ecgate:/remote/file/path /local/file/path

Avoiding storing SSH Keys

You may wish to avoid storing SSH Keys to disk and always (and only) use the SSH Agent.

tsh  client versions 7.1.0 onwards have this feature:

-k, --add-keys-to-agent        Controls how keys are handled. Valid values are [auto no yes only]. 

X11 under macOS

$ brew install --cask xquartz
# start xquartz app
$ export DISPLAY=:0
$ ssh -X ....

Legacy Configuration

For OpenSSH clients older than 7.3, the following configuration can be used:

# ~/.ssh/config file:
Host ecgate
  User ab0
  ProxyCommand /usr/bin/ssh -q -W %h:%p ab0@shell.ecmwf.int

Similar setup can be configured to use jump.ecmwf.int for Atos HPCF and other Bologna-based hosts.

Difficult client environments

You might not be able to download and run tsh, or access our web login service, or run the SSH Agent, from where you wish to use ssh.

Instead you can use (or copy) the identity files which tsh stores in $HOME:

# ~/.ssh/config file:
Host ecgate
  User ab0
  IdentityFile ~/.tsh/keys/shell.ecmwf.int/firstname.lastname@ecmwf.int
  ProxyCommand /usr/bin/ssh -q -i ~/.tsh/keys/shell.ecmwf.int/firstname.lastname@ecmwf.int -W %h:%p ab0@shell.ecmwf.int

This is a good way to access Teleport credentials via a shared file system from any host.

Similar setup can be configured to use jump.ecmwf.int for Atos HPCF and other Bologna-based hosts.

Even more difficult client environments

Recent Fedora Linux distributions (such as Fedora-33) using OpenSSH 8.4p1 no longer accept the "ssh-rsa" signature scheme using the SHA-1 hash algorithm in conjunction with the RSA public key algorithm.

As a workaround for this problem, you  may need to add ssh-rsa (rsa-sha2-256 or rsa-sha2-512 can also be used) as a PubkeyAcceptedKeyTypes to your ~/.ssh/config file:

# ~/.ssh/config file:
Host ecgate
  User          ab0
  PubkeyAcceptedKeyTypes +ssh-rsa
  IdentityFile	~/.tsh/keys/shell.ecmwf.int/firstname.lastname@ecmwf.int
  ProxyCommand /usr/bin/ssh -q -o PubkeyAcceptedKeyTypes=+ssh-rsa -i ~/.tsh/keys/shell.ecmwf.int/firstname.lastname@ecmwf.int -W %h:%p ab0@shell.ecmwf.int

Similar setup can be configured to use jump.ecmwf.int for Atos HPCF and other Bologna-based hosts.

Network Requirements

The service is configured to use only standard ports 22, 80, and 443, to help with access wherever users are.

Additional configuration at the local user site may be required to allow connections out. The diagram below shows the TCP ports and destination hosts used.

Versions 8.3.5 and newer of the tsh application support use of proxies (HTTP_PROXY , etc).  However, if the HTTPS_PROXY , etc, environment variable is set, tsh  will also attempt to make an ssh connection on port 22 using the proxy.  If your proxy does not support ssh connections to port 22 then set:

$ export no_proxy="shell.ecmwf.int:22,jump.ecmwf.int:22"
  • The "tsh login" step uses ports 80 and 443 in order to log in to the service and obtain the client certificate. 
    • initially it contacts shell.ecmwf.int at port 443 (user is able to see these steps by using tsh login --debug switch)
    • then it opens a local http client on a high port 64xxx, produces a link on the localhost for the user to follow (like http://127.0.0.1:64068/da92794b-9d41-4008-ae6f-83fb77f64486) and waits for a callback from shell.ecmwf.int
    • that localhost url then redirects user for OIDC authentication at https://accounts.ecmwf.int (port 443) involving Keycloak linked to user accounts on AD and HID token
    • upon successful authentication, tsh receives a callback from shell.ecmwf.int and receives the client certificate completing it's login workflow
    • from this point on, with the client certificate which is valid for 24 hours, user is authorised to access hosts behind the teleport proxy either via tsh ssh or OpenSSH workflows
  • Your ssh client uses standard port 22 for server access.
  • The web shell service also uses port 443 on the same host.

Gravitational Teleport ECMWF Implementation v1.0 Copy Copy



  • No labels