This tutorial is meant to support the training activity on the web API that may occur in the ECMWF's classroom.
The API is developed in pure python and the latest version is made available via Python Package Index (PyPI)
Installing the web API
The API installation within the ECMWF premises- On an classroom account
- On a mac/Linux notebook
- On a notebook with windows installed
Disabling the proxy settings
Retrieving data
Installing the web API on a classroom account
Enabling a virtual environment
Users do not have root access. In order to allow them to install python packages they have to setup a virtual environment.
Those with root access, or that have already configured a working python environment can go tho the next session.
To fully cover the topic of installing virtual environments, users can look at this tutorial Using virtual Python environment.
Users that want to permanently and quickly access the virtual environment have to create a file $HOME/.bashrc and put the following content in it.
# Use a sensible umask umask 022 # Place to keep your virtual environment export WORKON_HOME="$HOME/.virtualenvs" mkdir -p $WORKON_HOME # Use Python interpreter from /usr/local/apps python_home="/usr/local/apps/python/2.7.12-01" # Use our Python interpreter for virtualenvwrapper export VIRTUALENVWRAPPER_PYTHON="$python_home/bin/python" export VIRTUALENVWRAPPER_VIRTUALENV="$python_home/bin/virtualenv" export VIRTUALENVWRAPPER_VIRTUALENV_ARGS="--no-site-packages" . $python_home/bin/virtualenvwrapper.sh
Users that do not want to have the virtual environment to be available at each access they can use a different file e.g. use_environments.
The python version defined in the python_home environmental variable may change with time (new versions available).
To check which version are available: type module avail python and press enter.
To be able to use one specific version, just type module load [your choiche] and then check if the virtualenv command is available.
With the python ready to use and the script with the configurations to execute the virtualenv command, we have to source the script to configure our shell and then install a virtual environment.
source ~/.bashrc # (or source use_environments) mkvirtualenv cdsapi # or add -p [path to python executable you wish to use it can be different from the one specified in the set up script] workon cdsapi # to activate the environment / deactivate # to deactivate the environment
After entering the virtual environment using the command workon, we can use pip to install any package available via PyPI.
Installing the API using pip
Just execute:
pip install cdsapi
Or in case of root install sudo pip install cdsapi
Installing the API using conda
/<install_path>/conda/anaconda2/bin/conda config --add channels conda-forge /<install_path>/conda/anaconda2/bin/conda install cdsapi
For non linux/mac users
The point is to download and install your favorite python version then install the command pip, following common instructions, set up the command line environment to point to python and pip and then run pip install cdsapi.
Using the API
Installing credentials
Users have to locate theyr credentials on the climate data store.
In order to have credentials, users must register in the climate data store.
Registered users can access https://cds.climate.copernicus.eu/api-how-to and locate the field named "Install the CDS API key" .
The two lines specifying url and key must be copied in a specific file: $HOME/.cdsapirc
Non linux like users at the moment can use the credentials within the applications without using the file .cdsapirc (this means that they will produce code exposing the credentials which is a bad habit).
Disabling the ECMWF's proxy settings
For users using non ECMWF's computers (windows and other notebooks) this step is not necessary.
In order to use the web api we need that in the environment running it the proxy settings are deactivated (this is because the CDS is within the premises)
To achieve this goal, in the console with the cdsapi environment type:
export http_proxy= export https_proxy=
Just type enter after = !
Retrieving data
The 2 examples presented in https://cds.climate.copernicus.eu/api-how-to are good and simple examples working under unix like systems
#!/usr/bin/env python
import cdsapi
c = cdsapi.Client()
c.retrieve("reanalysis-era5-pressure-levels",
{
"variable": "temperature",
"pressure_level": "1000",
"product_type": "reanalysis",
"date": "2008-01-01",
"time": "12:00",
"format": "grib"
},
"download.grib")
#!/usr/bin/env python
import cdsapi
c = cdsapi.Client()
c.retrieve("insitu-glaciers-elevation-mass",
{
"variable": "elevation_change",
"format": "tgz"
},
"download.tar.gz")
In case the user is using python under windows, the script needs to be changed.
The change involves only the code line c = cdsapi.Client()
It may be changed in:
This method for providing the credentials may be considered bad practice, because a user will put in plane text his credentials.
A more safe way usable also in all platforms is to set up 2 environmental variables with the required informations:
CDSAPI_URL CDSAPI_KEY
More hands-on example:
#!/usr/bin/env python
import cdsapi
from datetime import datetime, timedelta
from dateutil.relativedelta import relativedelta
p_levels = [str(z) for z in ([1] + list(range(50, 1050, 50)))]
c = cdsapi.Client()
def days_of_month(y, m):
d0 = datetime(y, m, 1)
d1 = d0 + relativedelta(months=1)
out = list()
while d0 < d1:
out.append(d0.strftime('%Y-%m-%d'))
d0 += timedelta(days=1)
return out
for y in range(2008, 2018):
for m in range(1,13):
for d in days_of_month(y, m):
c.retrieve("reanalysis-era5-pressure-levels",
{
"variable": "temperature",
"pressure_level": p_levels,
"product_type": "reanalysis",
"date": d,
"time":[
'00:00','01:00','02:00',
'03:00','04:00','05:00',
'06:00','07:00','08:00',
'09:00','10:00','11:00',
'12:00','13:00','14:00',
'15:00','16:00','17:00',
'18:00','19:00','20:00',
'21:00','22:00','23:00'
],
"format": "netcdf"
},
"ea_t_{day}.nc".format(day=d)
)