...

Background

Based on user and developer experience of the Climate Data Store (CDS) Toolbox; the DSS offers the DSS JupyterHub Service as online computing environment and earthkit as the supported post-processing and visualisation software. JupyterLab sessions will be available to all DSS users (resource availability dependent) which provide fast access to data available on the various Data Stores and will allow users to perform post-processing and visualisation of this data. The sessions are considered small and not designed for large computation (see compute resource provisions table below). For larger computation task, DSS users should consider other JupyterHub resources, for example WEkEO.

The JupyterLab sessions provided by DSS include an interactive on-boarding tutorial. For further documentation regarding navigating your way around a JupyterLab session, please refer to the JupyterLab documentation pages.

How to access

The DSS JupyterHub Service is available from the ECMWF JupyterHub launcher page. Access requires ECMWF log in credentials (as required by the Data Store web-site to download data), including a two-factor authentication. Two-factor authentication can be setup by updating your ECMWF account credentials.

...

Environments available to DSS users

DSS users will be able to spawn sessions with the environment summarised in the table below. This can be selected from the "Select an Environment" dropdown selector on the JupyterHub Launcher. Please note that additional environment options may be added to this list as the Service evolves to meet the needs of users.

...

For reference, a month for one variable in the ERA5 hourly data on single levels is roughly 1.5 Gb. Larger volumes of data could be computed if using block-wise processing of data, e.g. using dask chunks in xarray.

Pre-installed software

Python environments

Default (earthkit) environment

The default Python environment is created using conda-forge with a python version 3.12.8 and the following environment.yml file below. This environment includes a number of ecmwf python packages, including the latest stable release of earthkit. This is the python environment used when launching the earthkit Notebooks and Consoles from the quick start menu.

name: base
channels:
  - conda-forge
dependencies:
  - python
  - pip
  - jupyterlab-git
  - ipywidgets
  - ipykernel
  - nodejs
  - git
  - yaml
  - pyyaml
  - beautifulsoup4
  - jupyter-server-proxy
  - numpy
  - pandas
  - xarray
  - numexpr
  - scipy
  - seaborn
  - dask
  - cartopy
  - shapely
  - plotly
  - netcdf4
  - cf-units
  - Markdown
  - toolz
  - tqdm
  - adjustText
  - aws-requests-auth
  - bokeh
  - voila
  - docstring_parser
  - filelock
  - metview-batch
  - metview-python
  - cdsapi
  - ecmwf-api-client
  - ecmwf-opendata
  - zarr
  - jupyterlab-tour
  - pip:
    - earthkit

CDO environment

The CDO software is available in a separate python environment. This can be used when selecting a Notebook or Console from the launcher tab, or if using a bash terminal the CDO environment can be activated with the following:

conda activate cdo

User installation.

You can install additional packages from the (open-source) conda-forge channel (`conda install PACKAGE-NAME`), or from PyPi (`pip install PACKAGE-NAME`). This packages will be installed in your local storage and will be available next time you create a session.

...

User storage

The DSS JupyterHub Service will offer two forms of storage. Please be aware that both of these options, and the way that they have been configured, are subject to change.

Private storage

Each DSS user will have a "home" Private Storage allocation (see table above for size). If you do not use the DSS JupyterHub Service for a period of 31 consecutive days the Private Storage will be removed. This storage is only accessible to you.

...

Scratch Storage

Each DSS user will have an allocated quota on the temporary scratch disk, i.e Scratch Storage (see table above for size). If you exceed the maximum quota, a clean up script will irreversibly remove your oldest files (by modified time). Should the DSS user circumvent the quota, in addition to any other rights available, we reserve the right to delete any files stored in the Scratch Storage of the DSS user.

The Scratch Storage is mounted on a scratch disk, which is a shared resource and is cleaned regularly to ensure that the disk does not exceed capacity. Therefore, the lifetime of the storage of files in the Scratch Storage depends on the overall usage of the DSS JupyterHub Service by all DSS users. The cleanup of files is based on the least recently modified files. This means that files stored here should not be considered permanently stored, they should exist for your current session and may or may not be there when you return. 

Shared resources

There is a shared resources directory available from the home in the folder titled notebooks-library. This ‘read only’ directory contains resources provided by Copernicus Climate Change (C3S) and Copernicus Atmospheric Monitoring (CAMS) and ECMWF, including but not limited, the notebooks found in the C3S training material. When using these notebooks, you can save them to your home directory and edit them as you wish. The notebooks library is managed as a git repository which is cloned each time a session is spawned, for reference the repository is here: https://github.com/ecmwf-projects/dss-jupyterhub-notebooks/  

External network access

SSH connections are disabled

The DSS JupyterHub Service sessions do not allow SSH connectivity for security reasons. Therefore, you must use the HTTPS address for any git repositories that you want to clone.

Right to suspend service

This Service is provided according the Terms of Use for the ECMWF Data Store Service JupyterHub. We reserve the right at any time and our sole discretion with or without prior notice to suspend the Service to DSS usersor terminate any DSS user’s access to the DSS JupyterHub Service,particularly in cases of violation of Terms of Use for the ECMWF Data Store Service JupyterHubor any of the applicable license terms or if a Private Storage is not accessed (i.e., by spawning a session) for a certain period of time beyond thirty-one (31) consecutive days.