This page describes the JupyterHub service provided by the ECMWF-Data Store Service (DSS), hereinafter referred to as DSS JupyterHub Service or Service, including how to access the service Service and what resources and software are available. For documentation regarding navigating your way around a JupyterLab session, please refer to the JupyterLab documentation pages.
...
| Note |
|---|
| Please note that by launching a JupyterHub session you are agreeing to the terms and conditions of use Terms of Use for the ECMWF Data Store Service JupyterHub. |
...
Based on user and developer experience of the Climate Data Store (CDS) Toolbox; the ECMWF- DSS offers a the DSS JupyterHub service Service as online computing environment and earthkit as the supported post-processing and visualisation software. JupyterLab sessions will be available to all DSS users (resource availability dependent) which provide fast access to data available on the various Data Stores and will allow users to perform post-processing and visualisation of this data. The sessions are considered small and not designed for very large computation (see compute resource provisions table below). For larger computation task, DSS users should consider other JupyterHub resources, for example WEkEO.
The JupyterLab sessions provided by ECMWF- DSS include an interactive on-boarding tutorial. For further documentation regarding navigating your way around a JupyterLab session, please refer to the the JupyterLab documentation pages.
How to access
The DSS JupyterHub will be Service is available from the ECMWF JupyterHub launcher page. Access requires ECMWF log in credentials (as required by the Data Store web-site to download data), including a two-factor authentication. Two-factor authentication can be setup by updating your ECMWF account credentials.
Once logged in, DSS users are given a choice of environment to use for their Jupyter DSS JupyterLab session from from a dropdown menu at the top of the page. The DSS offers a single environment to users: "ECMWF Data Store Service".
To start your session, click the "Start" button, if enough resources are available you will be assign a Jupyter DSS JupyterLab session which will be running for the length of time stated in the resource provisions table below.
...
The "Rollback" image is only available for a limited time and we encourage DSS users to update their workflows to use the "Default" image as soon as possible.
...
| Note | ||
|---|---|---|
| ||
All JupyterHub DSS JupyterLab sessions running on this service are time limited. When the time is up, the instance will be killed automatically along with any active processing that may be taking place. You can only have one session running. If you left one running, DSS JupyterHub will connect you straight back into it. If the service Service is busy and there are no resources available, you will be informed as such and will need to try again later. In the the early stages of the service we will closely monitor usage, and modify options to ensure that we provide a fair service to all our users.To ensure fair usage of DSS JupyterHub Service and/or the respective data store, ECMWF reserves the right to prioritise smaller (and shorter) sessions before others. |
| Info | ||
|---|---|---|
| ||
This is the general ECMWF JupyterHub launcher, therefore it is possible that you have access to more than the Data Store option described here. |
...
DSS users will be able to spawn sessions with the environment summarised in the table below. This can be selected from the "Select an Environment" dropdown selector on the JupyterHub Launcher. Please note that additional environment options may be added to this list as the service Service evolves to meet the needs of users.
...
Default (earthkit) environment
The default python Python environment is created using conda-forge with a python version version 3.12.8 and the following environment.yml file below. This environment includes a number of ecmwf python packages, including the latest stable release of earthkit. This is the python environment used when launching the earthkit Notebooks and Consoles from the quick start menu.
...
| Note | ||
|---|---|---|
| ||
As specified in article Article 5.5 of the Terms of Use for the ECMWF Data Store Service JupyterHub, it is the DSS user's responsibility to ensure they have all the necessary rights to use any of services, applications (including software), data and products used on DSS via the DSS JupyterHub Service. The software and configuration provided in the initial environment uses open-source channels only (i.e. conda-forge) and we encourage DSS users to use, and contribute to, open source software distributions. |
User storage
The DSS JupyterHub Service will offer two forms of storage for use in the JupyterHub. Please be aware that both of these options, and the way that they have been configured, are subject to change as the project develops.
| Storage type | Size | Longevity |
|---|---|---|
| Private storageStorage | 1 Gb | Permanent, if used every 31 days. |
| Scratch storageStorage | 100 Gb | Temporary, lifetime depends on overall usage |
Private storage
Each DSS user will have a "home" storage Private Storage allocation (see table above for size). If you do not use the DSS JupyterHub service Service for a period of 31 consecutive days the private storage Private Storage will be removed. This storage is only accessible to you.
The DSS service JupyterHub Service does not provide any back-up for the data stored, therefore we strongly advise that you use git repositories to back up any files stored in the private storage. This could be used to recreate any work should your private storage Private Storage be removeddeleted. JupyterHub JupyterLab provides a git plugin which makes it simple to clone your repository.
| Expand | ||
|---|---|---|
| ||
|
Scratch
...
Storage
Each DSS user will have an allocated quota on the temporary scratch disk, i.e Scratch Storage (see table above for size). If you exceed the maximum quota, a clean up script will will irreversibly remove your oldest files (by modified time). Any attempt to circumvent this behaviour is considered malicious and will lead to your access to JupyterHub being revoked.Should the DSS user circumvent the quota, in addition to any other rights available, we reserve the right to delete any files stored in the Scratch Storage of the DSS user.
The Scratch Storage is mounted on a scratch disk, which The scratch disk is a shared resource and is cleaned regularly . When the shared usage of all users exceeds the maximum quote, the files modified least recently will be removedto ensure that the disk does not exceed capacity. Therefore, the lifetime of the storage of files in the Scratch Storage depends on the overall usage of the DSS JupyterHub Service by all DSS users. The cleanup of files is based on the least recently modified files. This means that files stored here should not be considered permanently stored, they should exist for your current session and may or may not be there when you return. The lifetime of these files will depend on the general usage of the service, and at this stage it is not possible to provide an expected lifetime of such files.
...
Shared resources
DSS JupyterHub Service users have access to a library of shared notebooks resources. These can be found in folder titled notebooks-library in the home directory of the JupyterLab sessions. This read only directory contain resources provided by Copernicus and ECMWF. If using these notebooks, you can save them to your home directory and edit them as you wish. The notebooks library is managed as a git repository which is cloned each time a session is spawned, for reference the repository is here: https://github.com/ecmwf-projects/dss-jupyterhub-notebooks/
There is a shared resources directory available from the home in the folder titled notebooks-library. This read only ‘read only’ directory contain contains resources provided by Copernicus Climate Change (C3S) and Copernicus Atmospheric Monitoring (CAMS) and ECMWF. If , including but not limited, the notebooks found in the C3S training material. When using these notebooks, you can save them to your home directory and edit them as you wish. The notebooks library is managed as a git repository which is cloned each time a session is spawned, for reference the repository is here: https://github.com/ecmwf-projects/dss-jupyterhub-notebooks/
External network access
SSH connections are disabled
...
Right to suspend service
This service DSS JupyterHub Service is provided according the Terms of Use for the ECMWF Data Store Service JupyterHub. We reserve the right to suspend the service to users if we detect that terms and conditions are infringed. Suspension may be triggered automatically, and may only be reinstated when we have investigated the specific use case.