...

• Using Teleport
• Teleport Under the Hood

Security Context

We have remote users, who we do not know very much about (e.g. researcher in a university who is sponsored by a Member State national weather service), connecting to our HPC cluster on which we also run mission-critical operational forecasting. As these aspects are unavoidable, measures need to be in place to sufficiently secure the access to our HPC facility.

Let's take the JASMIN cluster service as an example and compare our situation to theirs.

• JASMIN is a research facility, so does not have the same availability and integrity concerns as ECMWF.
• We can see that it's normal for users to have to go through a non-trivial process to access a cluster: generating an SSH keypair and uploading the public key to a website. This also implies ECMWF running such a service to manage key uploads.
• JASMIN "requires" users to secure their key with a passphrase, but there's no way this can be enforced. It's quite likely users don't bother, and the SSH keys are copied around freely and even lost and replaced, while remaining active for access to the facility. 
• We know from other remote cluster providers that it's common to have to connect to a bastion host, and then hop onwards to the final working node. At ECMWF we much prefer users have a single hop straight to the cluster.
• Although our ActivID tokens work well, they are very expensive and the software (at ECMWF's end) is unreliable and troublesome to maintain. We would like to move away from ActivID in the post-BOND timeframe, so any access service should not be bound too tightly to it.

For these reasons, Teleport is a good choice, as it allows separation from ActivID, avoids a complex initiation for the user, avoids maintenance of SSH keys and their inherent security risks, avoids ECMWF running additional web services, and is single-hop client to server.

Understanding ecAccess

Downloading tsh 

The tsh application is required to perform the single sign-on step once per day.

As tsh is written in Go, it is very portable and has minimal dependencies.

• Download from the Gravitational website and place into your $PATH 

The binary is available for Linux 32-bit, 64-bit, and ARM, as well as MacOS and Windows 64-bit.

Initial login

Run tsh, giving the location of our Teleport gatewayA user outside of ECMWF wishes to ssh to one of the Centre's clusters, batch farms, or workstations. This can be done by running:

ssh sy0@ecaccesstsh login --proxy=tele.ecmwf.int

The user is prompted for a password, which happens to be the one-time code from an ActivID token.

After authenticating to ecAccess, the user is prompted for the destination host:

[cca-login,ecgate]
?

In fact, the user can enter any ECMWF server or workstation name.

The destination server or workstation must run the ecauth binary which is installed "setuid root" (so it can become any user).

ecauth receives the TCP/IP connection from ecAccess and spawns a shell as the original user (sy0, in this example).

Understanding our SSH Proxy service

When servers or workstations inside the Centre wish to ssh to the Internet (not the RMDCN), we tell the client to use our SSH Proxy service.

The Proxy sits in the DMZ and allows transit of the TCP/IP connection from ssh client to server. It does not interrupt the encryption in any way.

Clients inside the Centre use the openssh ProxyCommand setting to direct the connection via our Proxy:

ProxyCommand /usr/bin/connect -H proxy.ecmwf.int:2222 %h %p

This is a commonly used behaviour in ssh, and in fact new openssh clients and servers support the ProxyJump setting, to simplify configuration even further.

Understanding Teleport

Teleport functions somewhat like a combination of ecAccess, and the SSH Proxy service, with additional features relating to role-based access control and single sign-on.

Users outside of the Centre configure their clients to use Teleport as a proxy service for the destination host. That is, it's a bit like the reverse of systems inside ECMWF connecting to the Internet.

Like ecAccess, Teleport can have its own binary installed on the destination server or workstation. However, this is optional, and standard openssh server works too.

The notable difference between Teleport and our current services is that a specific command-line application, called tsh, is required to do the single sign-on function (once per day).

The single sign-on step will open a web browser and prompt the user for their username and password (from the token). This means if we migrate away from ActivID in the future then the Teleport service is not affected in any way, because login is delegated to the web service.

After login, a user's ssh-agent is populated with some SSH Certificates signed by the Teleport server, valid for a limited time (say, eight hours).

SSH clients present the certificates in a similar way to SSH Keys each time they login, and the certificates are validated by both the Teleport gateway and the destination host.

Your default web browser will open and you should login with your email address and token code workstation password.

To see the credentials that tsh was given, their attributes and remaining lifetime, run:

$ tsh status
> Profile URL: https://tele.ecmwf.int:3080
  Logged in as: oliver.gorwits@ecmwf.int
  Cluster: tele
  Roles: *
  Logins: sy0
  Valid until: 2019-11-13 17:10:03 +0000 GMT [valid for 55m0s]
  Extensions: permit-X11-forwarding, permit-agent-forwarding, permit-port-forwarding, permit-pty

Configuring servers and workstations for Teleport

At the destination host end, openssh needs to be configured to trust the SSH Certificate that your client is passing.

This can be done at the server level for all users, or by an individual user within their own account.

A user can add the following to their ~/.ssh/authorized_keys file:

cert-authority ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN85frMTtRzQaVjkHGM3NTOJD/5awWn2i1sAofzlO0PXM1jX/H+4zQKn1+ATUqU+TTU2v3V7fhZm0cWqRrofSLDVC80FkCFqzZRq2E4kMBP5sx4yf/mBKLzJ6luE8eV/3W0V6KEch5TV2ON8ltwFPWjB3D/puPU010UOJH/arlpW5h+n9dAtBCAtVuMYBEz/5uWcKcTJkFe2usQTVmpiEmt/yAx4LfLTFPs2+izLo+N3dBW92HFnKnQftZI5s8ysOENbKmxdWfcxdID5E91oneAvkmEKBeUutlYY3GV9621iuKnlEw6WF8wfQapdozzhSyb+LhqPDiBRotnerhA69/ clustername=tele&type=user

A server administrator can place the same key into /etc/ssh/teleport-user-ca.pub

They would then add the following to their /etc/ssh/sshd_config  file:

AuthorizedKeysFile /etc/ssh/teleport-user-ca.pub .ssh/authorized_keys .ssh/authorized_keys2

Connecting to hosts through Teleport

Your ssh client will need to know about the Teleport gateway (proxy).

For OpenSSH 7.3 or newer, add the following to your configuration (~/.ssh/config):

Host ecmwf-hpcf
  HostName  hpcf-login.ecmwf.int
  ProxyJump tele.ecmwf.int:3023
  ForwardX11 yes

For OpenSSH clients up to 7.2, the following will work instead:

Host ecmwf-hpcf
  HostName hpcf-login.ecmwf.int
  ProxyCommand /usr/bin/ssh -q -p 3023 -W %h:%p tele.ecmwf.int
  ForwardX11 yes

Then, you can connect to the destination host:

ssh ecmwf-hpcf

SCP, X11 and Port Forwarding

scp, X11, and port forwarding will all work through the Teleport gateway.

Of course, such features also need to be permitted on the destination server or workstation.