Access Morpheus portal
Login to the Morpheus portal according to the provided informations.
Reference: Logging in to Morpheus
User Settings
As first step after login it is needed to finalize the setup of the user settings.
ssh keys settings
It is required to setup the Linux ssh public key by following the instructions in the page: Add your SSH key pair
the key will be uploaded to the provisioned Linux Virtual Machines and the user can connect to it via ssh.
In order to add your public key go under Menu→ Infrastructure → Trust and then press the +ADD button and alternatively:
- select "Existing Key Pair" if you you have already your keys pairs (private and public keys) and you want just to upload your public key (only public key is needed)
or
- Select "Generate Key Pair" if you want to let Morpheus to generate a key pair (private and public keys) for you. In this case save the generated private key.
In order to distinguish the keys from other users of the tenancy, assign the following name : <your username>_key
Example for "Existing Key Pair" uploading:
Linux user settings
After setting up the key it is needed to finalize the setup of the Linux user settings by following the instructions in: Setting up your User Settings
Access the User Settings tab by clicking on the User in the top-right corner and then select "User Settings" :
Within the User Settings then configure the "Linux Settings" tab that is required for creating the user on the created virtual machines, set the defined password and set the public key (note that only ssh key authentication is allowed).
Provision a new instance (Linux Virtual Machine)
You can provision a new instance in accordance to the instructions on the page: Provision a new instance - web
Within this tutorial it is provisioned a Rocky Linux 9.2 virtual machine with the "rocky-9.2-data" layout that includes a set of preinstalled software.
Access the Menu→ Provisioning → Instances tab and click the +Add button
Select the Rocky Linux instance type
Configure the instance
Assign a name, environment and label:
Suggestions:
- Name : <your username>-vm
- Environment: Test
- Label: training
and then click NEXT.
Configure the VM version, layout, plan, network, security group :
- Version : 9.2
- Layout: rocky-9.2-data
- Plan: 4cpu-4gbmem-30gbdisk
- Networks: external-internet
- Security Group: ssh-https
- Others: leave defaults
( A description of these fields can be find in the documentation page: Provision a new instance - web )
and then click NEXT.
Leave default setting for Automation, Backups and Lifecycle
and then click NEXT.
Review settings and complete
Review the settings and if everything is as expected then press Complete button to start the instance creation.
Monitor the deployment
From the Instances page it is possible to monitor the status of the VM creation.
and by clicking on the Instance it is possible to check more details about the progress.
Deployment completed - Instance running
Once the deployment is finalized (typically after few minutes), the instance appears as Running (green icon)on the Morpheus interface:
and it is possible to check more details on the instance page:
Access to the created instance
According to the applied configuration, it was selected the "external-internet" network, which gives the VM a public IP to which is possible to connect directly via ssh.
The assigned public IP can be reviewed from the Morpheus interface:
once taken note of the public IP then it is possible to connect via ssh.
Assuming you have an ssh-agent with your stored key then it would be enough:
ssh <your username>@<public IP>
otherwise you can specify your private key in the connection :
ssh -i ./path/to/private_key <your username>@<public IP>
S3 Object Storage
The European Weather Cloud offers the possibility to use a S3 Object Storage : Object storage: How to use s3cmd and s3fs
A common way to manage the S3 object storage is via the "s3cmd" command line tool that is pre-installed within the instance.
Configure s3cmd client
You will need to configure s3cmd before you can use it. The tool will read the configuration on ~/.s3cfg
Create the configuration file if it does not exist:
touch ~/.s3cfg
Edit the file and set up at least the following parameters (replace the fields with the provided endpoint and access/secret keys).
host_base = <endpoint> host_bucket = access_key = <youraccesskey> secret_key = <yoursecretkey> use_https = True
Retrieve a file from S3 Object Storage
The following instructions explain how to list the files within the bucket and retrieve the file locally using the "s3cmd" tool.
1) List files in the existing bucket:
s3cmd ls s3://mybucket/
2) Retrieve file from the S3 bucket locally:
s3cmd get s3://mybucket/file.grb
3) Verify the file has been copied locally:
ls -l file.grb
Explore the created ECMWF "data layout" instance
The ECMWF data layout instance comes with a basic set of pre-installed tool that allows users to start retrieving and working with the ECMWF data on the cloud.
Specifically the ECMWF data layouts include:
- a MARS client with an optimized data access, similarly of what present on the ECMWF HPCF
- a pre-installed conda environment with main ECMWF packages (e.g. ecCodes)
General overview
Verify your user has sudo rights
In order to get sudo rights you'll need to insert the password configured under the "Linux Settings" section of the "User Settings".
[traininguser@traininguser-vm ~]$ sudo -i
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
[sudo] password for traininguser:
[root@traininguser-vm ~]#
Verify your VM cpu and ram resources
You can verify that your VM has 4 processors according to the selected plan.
[root@traininguser-vm ~]# cat /proc/cpuinfo | grep -E '^processor' processor : 0 processor : 1 processor : 2 processor : 3
You can verify that the VM has 4GB RAM according to the selected plan.
[root@traininguser-vm ~]# cat /proc/meminfo | grep -E '^MemTotal' MemTotal: 4011760 kB
Verify your VM disk space
Verify the Disk space is 30GB according to the selected plan.
[root@traininguser-vm ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 784M 8.6M 776M 2% /run /dev/vda1 29G 5.1G 22G 19% / tmpfs 392M 0 392M 0% /run/user/1000
Test a new system package installation
[root@traininguser-vm ~]# dnf install htop Last metadata expiration check: 4:08:03 ago on Mon Oct 30 12:29:56 2023. Dependencies resolved. ===================================================================================================================================== Package Architecture Version Repository Size ===================================================================================================================================== Installing: htop x86_64 3.2.2-1.el9 epel 183 k Transaction Summary ===================================================================================================================================== Install 1 Package Total download size: 183 k Installed size: 440 k Is this ok [y/N]: y Downloading Packages: htop-3.2.2-1.el9.x86_64.rpm 1.6 MB/s | 183 kB 00:00 ------------------------------------------------------------------------------------------------------------------------------------- Total 1.5 MB/s | 183 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : htop-3.2.2-1.el9.x86_64 1/1 Running scriptlet: htop-3.2.2-1.el9.x86_64 1/1 Verifying : htop-3.2.2-1.el9.x86_64 1/1 Installed: htop-3.2.2-1.el9.x86_64 Complete!
Run the "htop" command provided by the just installed package:
[root@traininguser-vm ~]# htop
Exit from "htop" view (ctrl+c or F10).
If still impersonating "root", return to the normal user:
[root@trainingadmin-vm ~]# exit logout [tenantadmin@trainingadmin-vm ~]$
Configure your MARS client
The "data" layout includes the MARS client that just needs to be configured to start retrieving data from the archive.
The configuration steps are described in the Knowledge Base dedicated page .
Configure the email and token
- In your web browser...
- Go to https://apps.ecmwf.int/v1/key
- Log in with your ECMWF user and password if you are not logged int already
- Copy the contents of .ecmwfapirc, containing your email address and key.
In your VM, create a file in your home directory called ~/.ecmwfapirc and paste the contents previously copied from the web.
Run a sample MARS retrieval
Create a text file "mars_request.req" containing your request:
[traininguser@traininguser-vm ~]$ cat <<EOF >mars_request.req retrieve, class = od, stream = oper, type = an, expver = 1, date = 20231015, time = 12, param = 130.128, levtype = pressure level, levelist = 1000/850/700/500, target = "output.grb" EOF
then submit the request as follows :
$ mars mars_request.req
Test conda environment and pre-installed packages
Activate base conda environment
[traininguser@traininguser-vm ~]$ conda activate (base) [traininguser@traininguser-vm ~]$
List available packages within the conda environment
[traininguser@traininguser-vm ~]$ conda list # packages in environment at /opt/conda: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge alsa-lib 1.2.10 hd590300_0 conda-forge attr 2.5.1 h166bdaf_1 conda-forge bison 3.8.2 h59595ed_0 conda-forge blosc 1.21.5 h0f2a231_0 conda-forge boltons 23.0.0 pyhd8ed1ab_0 conda-forge boto3 1.28.73 pypi_0 pypi botocore 1.31.73 pypi_0 pypi brotli-python 1.1.0 py310hc6cd4ac_0 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.19.1 hd590300_0 conda-forge ca-certificates 2023.7.22 hbcca054_0 conda-forge .....
Test ecCodes available in the pre-installed conda
Test ecCodes is installed by checking the number of messages contained in the grib file downloaded in the previous MARS exercise.
First activate base conda environment:
[traininguser@traininguser-vm ~]$ conda activate (base) [traininguser@traininguser-vm ~]$
then use the "grib_ls" command provided by ecCodes:
(base) [traininguser@traininguser-vm ~]$ grib_ls output.grb output.grb edition centre typeOfLevel level dataDate stepRange dataType shortName packingType gridType 1 ecmf isobaricInhPa 1000 20231015 0 an t spectral_complex sh 1 ecmf isobaricInhPa 850 20231015 0 an t spectral_complex sh 1 ecmf isobaricInhPa 700 20231015 0 an t spectral_complex sh 1 ecmf isobaricInhPa 500 20231015 0 an t spectral_complex sh 4 of 4 messages in output.grb 4 of 4 total messages in 1 files
Delete the provisioned instance
Once reached the end of the tutorial you can delete the provisioned instance.
You can access the instance page and then press the button "DELETE".
You will be asked to confirm the operation. After confirming, it will start the deletion process.
