...

Based on user and developer experience of the Climate Data Store (CDS) Toolbox; the DSS offers a JupyterHub service as online computing environment and earthkit as the supported post-processing and visualisation software. Jupyterhub sessions will be available to all DSS users (resource availability dependent) which provide fast access to data available on the various Data Stores and will allow users to perform post-processing and visualisation of this data. The sessions are considered small and not designed for very large computation (see compute resource provisions table below). For larger computation task, users should consider other JupyterHub resources, for example WEkEO.

How to access

The DSS JupyterHub will be available from the ECMWF JupyterHub launcher page, linked from the Data Store web-sites. Access requires ECMWF log in credentials (as required by the Data Store web-site to download data), including a two-factor authentication. Two-factor authentication can be setup by updating your ECMWF account credentials.

Once logged in, users are given a choice of environment to use for their Jupyter session session from a dropdown menu, with several additional option depending on which environment you have selected. at the top of the page. The DSS offers a single environment to users: "ECMWF Data Store Service".

Just above the "Start" button there is a "Version" dropdown which refers to the version of the environment to use in your session. To ensure that the software is kept up-to-date, the version will be updated several times a year. We only support the "Default" environment version, we offer a 

...

...

Image Added

Image RemovedFigure1: Selecting DSS from ECMWF JupyterHub launcher page

...

DSS users will be able to spawn sessions with the environment summarised in the table below. This can be selected from the "Select an Environment" dropdown selector on the JupyterHub Launcher. Please note that additional environment options may be added to this list as the service evolves to meet the needs of users.

...

For reference, a month for one variable in the ERA5 hourly data on single levels is roughly 1.5 Gb. Larger volumes of data could be computed if using block-wise processing of data, e.g. using dask chunks in xarray.

...

...

.

...

Pre-installed software

Python environments

Default (earthkit) environment

The default python environment is created using conda-forge with a python version versoin 3.1112.10 8 and the following environment.yml file :below. This environment includes a number of ecmwf python packages, including the latest stable release of earthkit. This is the python enviornment used when launching the earthkit Notebooks and Consoles from the quick start menu.

name: base
channels:
  - conda-forge
dependencies:
  - python
  - pip
  - jupyterlab-git
  - ipywidgets
  - ipykernel
  - nodejs
  - git
  - yaml
  - pyyaml
  - beautifulsoup4
  - jupyter-server-proxy
  - numpy
  - pandas
  - xarray
  - numexpr
  - scipy
  - seaborn
  - dask
  - cartopy
  - shapely
  - plotly
  - netcdf4
  - cf-units
  - Markdown
  - toolz
  - tqdm
  - adjustText
  - aws-requests-auth
  - bokeh
  - voila
  - docstring_parser
  - filelock
  - metview-batch
  - metview-python
  - cdsapi
  - ecmwf-api-client
  - ecmwf-opendata
  - zarr
  - jupyterlab-tour
  - pip:
    - earthkit

CDO environment

To ensure the requirements are correctly installed, the CDO software is available in a separate python environment. This can be used when selecting a Notebook or Console from the launcher tab, or if using a bash terminal the CDO environment can be activated with the following:

conda activate cdo

User installation.

You can install additional packages from the (open-source) conda-forge channel (`conda install PACKAGE-NAME`), or from PyPi (`pip install PACKAGE-NAME`). This packages will be installed in your local storage and will be available next time you create a session.

User storage

The DSS will offer two forms of storage for use in the JupyterHub. Please be aware that both of these options, and the way that they have been configured, are subject to change as the project develops

...