Introduction

Users are responsible for the security of the virtualised resources under their control. This includes, but it is not limited to: virtual machines, network configuration, user accounts, disk volumes. 

Here are some security guidelines to help you secure your resources on the European Weather Cloud.

Network 

It is very important to keep your network configuration as secure as possible, as it is the gate any intruder will use to enter in your system. It is relatively simple to apply some good practices that will give a good extra security layer. Here below few strategies are advised.

Prevent external parties from connecting to your machines

Avoid using public IPs where practical

A VM only needs a public IP if you're expecting someone to connect to it from outside your tenancy.  You can avoid adding the public network or assigning a floating IP when you create a VM, and keep most VMs only on the private/internal network.

Additionally, It is a good idea to minimise the number of IPv4 addresses in use, as they are in short supply.

To reach VMs with only a private IP from outside, you first connect to a well-secured gateway machine with a public IP, then connect onwards internally to the target VM on the private network.

EWC provides an SSH proxy template to act as such a gateway.  Please see SSH Proxy (or SSH Bastion) for examples on how to use this to connect to internal VMs and resources.

Restrictive firewall (allow-listing)

Every virtual machine running in EWC comes with two firewalls: the firewall of the virtual machine itself (netfilter/iptables/nftables/firewalld) and the (Openstack) Security Groups. The machine firewall can be changed by admin users on the machine itself, while security groups must be configured by someone with Morpheus or Openstack access.  Which is most appropriate depends on whether you want to delegate to machine admins or wish to impose constraints on them.  In any event, both firewalls will apply at all times and network packets must be allowed by both in order to get through.

To be maximally secure, your virtual machine instances should be configured so that they allow the minimum required access needed to run your application - i.e. only open the ports that need to be opened and only open them for the least amount of IPs possible.. By default, virtual machines on EWC come with no security groups (which means all internal and external connections are blocked).  In order to connect to them, or to provide any kind of service, access has to be explicitly added by enabling some security groups and potentially modifying or disabling the local firewall. 

There are default Security Groups which are adequate for many situations and environments, please use those whenever possible. You may also create your own as described in Creating Security Groups in Morpheus.

Disable unneeded services

You should remove any software that you are not going to use with apt remove or dnf remove where appropriate. Do not run unnecessary services on your VM, especially those that are accessible from the outside. The more services you run, the more potential attack surfaces you have that top intruders might exploit. 

You can check what services are enabled with:

sudo systemctl status

And disable them with:

sudo systemctl disable <service>

You may also use the following command to identify any open ports:

netstat -tulpn

Use secure protocols

Wherever possible, use encrypted and secure communication protocols to avoid man in the middle attacks; this is when someone is able to monitor your communications and can read the data going through them, for example if you're using a public WIFI hotspot or have a malicious network administrator. For example: do not use HTTP, use instead HTTPS. Do not use FTP to transfer files, use instead FTPS or SFTP.

Avoid exposing certain services, such as databases or other internal management systems to the Internet. If you must access to them from outside the tenancy, consider tunneling them over SSH.

Use intrusion detection software

Tools such as denyhosts or Fail2ban can be used to analyse log files and ban IP addresses that are attempting to make brute-force attacks to your application. They are very powerful tools, but they have to be used used with care as they can lead to false positives, i.e. Banning IPs that should not be banned. These tools are a best practice to provide 24/7 services, while may not be necessary for single user VMs.

The images provided by EWC come with fail2ban preconfigured and active. See the EWC Fail2ban for more information on how to use this tool.

Software

Running secure software is also very important. It is not a trivial task to develop fully secure software, but there are some simple strategies that will help with the task.

Automatic software updates

The user is responsible for what's running on their machine and must plan and implement updates according to the risks and their needs. We can only recommend the following:

• The images provided by EWC come with automatic updates enabled for the security repositories only. This is made possible by leveraging dnf-automatic  for rpm based distros and unattended-upgrades for deb based ones.  The user can disable these or enable updates for all packages.
• Some updates, such as kernel upgrades, require manually rebooting the virtual machines. Please schedule this into your regular maintenance.
• If your use case does not support automatic updates, which is common for highly available setups, please make sure to schedule regular maintenance windows where the software upgrade is scheduled.
• If you're running public services, subscribe to security announcements for any software not covered by automatic updates, if there is a security problem that requires your action, you need to find it out as soon as possible.
• Do not rely on obscurity ("no one will notice this old bit of software..") - there are malicious people running a large number of automated scanners that continuously search the internet for vulnerable software or weak passwords.  A typical SSH server on the internet will be hit thousands of times a week with automated login attempts.

Only install from reputable sources

Be mindful of the sources for the software you install. Only install software from reputable sources. If possible, use the distribution's package manager (yum, dnf, apt, ...). Packages managers make it easy to install software, keep it updated, and uninstall it. If the desired software is not available in the distribution package manager repository, an official source must be used. Follow the instructions on the official website of the software you need. If more than one source is offered, think about using the one that provides an easier life-cycle (install/update/uninstall/...), like conda, snap or flatpak.

Be mindful about the user accounts in the VM

Keep an eye on the user accounts enabled in your system. Some applications create default accounts which are unnecessary or even directly insecure. An ideal scenario might be three accounts:

• root with ssh disabled and no password. This is the default in the images provided on ECMWF Cloud for the different OS (i.e. Ubuntu, Rocky,..).
• A user account for a sysadmin that can only be accessed via ssh keys and has sudo access.
  • ECMWF Cloud images provide this user pre-configured as well as part of cloud-init, the name of the user depends on the distribution (cloud-user, ubuntu), see the documentation for more information.
  • EUM tenancies typically provide user ids in LDAP that have this capability
• If running services, add a user account that run a single service and have no login possible, neither remote nor local access.

For general user accounts, consider disabling password login and use SSH keys instead.  When using SSH keys, challenge-response authentication is used instead. This means that for each login a different challenge is asked and a different response is the correct one. No secret (password or key) ever travels across the network.  If you have to use passwords, be aware:

• Passwords can be, with enough time and compute power, guessed with brute force
• More commonly, people use weak (easily guessable) or previously used / known passwords (https://haveibeenpwned.com/Passwords) - try to ensure passwords are complex and unique.  EWC default setups will warn about weak passwords.
• It's easiest if you use a password manager (e.g. https://keepassxc.org/) to generate and store passwords.

For publicly exposed services, prefer longer/less common usernames (e.g. not "admin") to reduce guessability - to login, the attacker must guess the combination of password and username.  You may not be able to control your users selecting weak passwords, but you can make the usernames more complex when adding them.

Password protect your SSH keys and make sure your key never leaves your control (ideally never leaves the machine where it was created, by using tools such as ssh-agent).

• if a key is somehow accidentally exposed (one common way is unintentionally including it in a github repository!), retire it and generate a new one.  There are brute forcing scanners that use lists of known keys (e.g. https://github.com/HynekPetrak/sshame)

Keep logs of your applications

Use the best practices for logging:

• Make sure that the services are logging to a secure location, that is as tamper-proof as possible.
• Keep the logs for a reasonably long amount of time.
• Consider logging to a remote server as well.

More information

If you are interested to learn more about security in cloud applications, we advise reading the material provided by NeCTAR. 

Acknowledgements

The EWC Team would like to acknowledge the following source of information for this page: 

• https://wiki.u-gov.it/confluence/display/SCAIUS/General+Security+Guidelines+for+ADA+cloud
• https://docs.csc.fi/cloud/pouta/security/