Climate Data Store (CDS) documentation

Last modified on Oct 25, 2024 15:51

Introduction

The Climate Data Store (CDS) is the cornerstone infrastructure which supports the implementation of the Copernicus Climate Change Service (C3S). It enables the provision of Essential Climate Variables (ECVs), climate analyses, reanalyses, projections and indicators at temporal and spatial scales relevant to adaptation and mitigation strategies for various sectoral and societal benefit areas.

The CDS is designed as a distributed system which provides improved access to local and remote datasets via a powerful service-oriented architecture.

The CDS offers seamless web-based and API-based search and retrieve facilities to access climate data and information. In addition, the CDS also provides a generic software toolbox that allows users to develop web-based applications that make use of the datasets available in the CDS.

The data provided by the CDS are free and open data, subject to the user agreeing to the relevant dataset licence(s). For further details, please see this article from the ECMWF Newsletter 151.

How the CDS works

The CDS provides a Catalogue which lists C3S data and products, including observations, reanalyses, seasonal forecasts and climate projections. Users can search through the catalogue, and can filter the entries by means of both faceted search (Product type, Variable domain, Spatial coverage, or Temporal coverage) and textual search. 

Users can request data from the CDS using a variety of methods:

• the CDS web interface
• the CDS Application Programming Interface (API)

1) The CDS web interface is an interactive system: the user fills a web form to construct their valid query and can then choose between three options:

• submit the form and download the result of the query;
• show the query as an API request;

Please make sure that you select all the mandatory fields before submitting a request.

This is shown in the following figures:

2) The CDS API is a service providing programmatic access to CDS data in Python (using the CDS API client). Users need to have a CDS account to use it with their related CDS API credentials. For a full description and some useful examples, please see How to use the CDS API. 

If users have more than one CDS API key (e.g. you also have a Atmosphere Data Store (ADS) account), you have to pass credentials in explicitly and the following method can be used:

import cdsapi
import yaml

with open('/path/to/ads/cdsapirc', 'r') as f:
        credentials = yaml.safe_load(f)
c = cdsapi.Client(url=credentials['url'], key=credentials['key'])

c.retrieve("dataset-short-name", 
           {... sub-selection request ...}, 
           "target-file")

We strongly suggest to construct CDS API requests by using the CDS web interface of the relevant dataset and using the 'Show API request' button to get the code.

Users can also set the PROXY within the CDS API script:

import requests
import cdsapi


session = requests.Session()

session.proxies = {
   'http': 'http://10.10.10.10:8000',
   'https': 'http://10.10.10.10:8000',
}

client = cdsapi.Client(session=session)

As the CDS is a distributed platform, CDS datasets are hosted at a number of different locations. Some data are stored on the CDS disks; others are hosted on the data providers' storage systems, e.g. the ECMWF Meteorological Archival and Retrieval System (MARS).

Table 2 presents the ERA5 family datasets which are available on CDS disks ('MARS internal)'. Table 3 shows the CDS datasets which are stored in the tape library (referred to as 'MARS external'). The rest of the datasets are stored in the data providers archives.

The way the user selects the items to include in a single CDS request can make a significant difference in terms of performance. Please see the page Efficiency tips for how to build an efficient request.

On the CDS, requests and relevant results are generally cached and the period depends on the user load on the CDS system. Thereafter, files are deleted starting from the oldest. Please note that cached data is used to fulfil a CDS request only if a given CDS request is exactly identical to a previous one.

Request states

After sending a request, the user can track its state on the 'Your Requests' page of the CDS web interface. Users can also check the status of the CDS system at the live page. There are five different states of a request:

1.  Accepted. Each request is assigned a unique ID and a priority. The priority is chosen according to different criteria, such as the origin of the request (CDS web interface/API/Toolbox). For example, the CDS web interface usually has higher priority because it is an interactive application and users expect an immediate response to their request..
2. In progress. The request is being fulfilled and the data is being collected from the archive.
   
3. Failed. The request encountered problems.
   
4. Unavailable. The data has expired from cache and therefore cannot be retrieved at the current time. In this case the request should be resubmitted.
   
5. Succeded. The resulting data file is ready to download.
   

User can check the live status of the CDS system here https://cds.climate.copernicus.eu/live

Limits

Limits are set on usage of CDS resources to ensure an appropriate level of performance for users.

Please note that these limits are changed from time to time according to the current workload of the system and number of concurrent tasks.

The CDS will queue requests which would otherwise cause any of these limits to be exceeded.

When a user CDS web request exceeds the number of fields limit, an information message appears at the bottom of the web interface page. 

Figure 4: Information message when a user CDS web request exceeds the number of fields limit.

Please note, that these limits are also enforced for requests sent via the CDS API.

Datasets

The tables below summarise the number of fields and volume size limits, as well as the major features of all CDS datasets. Table 2 shows the datasets stored in 'MARS internal', whereas Table 3 presents the datasets stored in 'MARS external'. The rest of the CDS datasets are stored in the data providers archives.

Data hosted on the CDS/'MARS internal'

Data hosted on 'MARS external'

Efficiency tips

1. Where the data are actually  stored can make a significant difference in performance. CDS data hosted in 'MARS internal' is stored on CDS disks, and so is faster to retrieve. The  'MARS external' datasets are stored in the ECMWF MARS (tape) archive, and in this case it is important to request as much data as possible from the same tape file in your CDS request.
   
2. Submit small requests over very large and heavy requests. This will ensure your requests are not penalised in the CDS request queue. In particular:

   1. For ERA5 data requests, please see: How to download ERA5. The first example in CDS API examples section below shows how to efficiently download a whole year of hourly data for 2m temperature (grib format, for an area subset) from the CDS, by asking for one month of data per request.

   2. For C3S seasonal forecast data requests, please see: Recommendations and efficiency tips for C3S seasonal forecast datasets.
   3. For UERRA data requests, please see: UERRA retrieval efficiency.
3. When using the CDS API or the CDS Toolbox, it is advised that users take as an example the API request or toolbox request script shown at the bottom of the CDS web  'Download data' page for the dataset of interest, and use this as the basis for your request.
4. Some ERA5 datasets, such as reanalysis-era5-complete, do not appear in the CDS catalogue on the web interface, but users can still retrieve the data through the CDS API. In these cases, users can make use of a subset of  the  ECMWF MARS keywords in their CDS API request, and should also follow the MARS efficiency rule of thumb (the idea is to request as much data as possible from the same tape file or to reduce the number of tapes involved.).
5. Users are strongly advised to check the allowed and not allowed keywords from the relevant dataset webform.

CDS API examples

CDS API Request for hourly data of one variable looping though months and years ( e.g 2m temperature for all months in 2018 and 2019)

import cdsapi

client = cdsapi.Client()
dataset = "reanalysis-era5-single-levels"

first_year = 2018
last_year = 2019

for year in range(first_year, last_year + 1):
    for month in range(1, 13):
        print("=========================================================")
        print("Downloading {year}-{month:02d}".format(year=year, month=month))
        request = {
                "product_type": "reanalysis",
                "variable": "2m_temperature",
                "year": str(year),
                "month": "{month:02d}".format(month=month),
                "day": [
                    "01", "02", "03",
                    "04", "05", "06",
                    "07", "08", "09",
                    "10", "11", "12",
                    "13", "14", "15",
                    "16", "17", "18",
                    "19", "20", "21",
                    "22", "23", "24",
                    "25", "26", "27",
                    "28", "29", "30",
                    "31"
                ],
                "time": [
                    "00:00", "01:00", "02:00",
                    "03:00", "04:00", "05:00",
                    "06:00", "07:00", "08:00",
                    "09:00", "10:00", "11:00",
                    "12:00", "13:00", "14:00",
                    '15:00', '16:00', "17:00",
                    "18:00", "19:00", "20:00",
                    "21:00", "22:00", "23:00",
                ],
                "area": [90, 170, 80,180,],
                "data_format": 'grib',
                "download_format": "unarchived"
            }
        target = "data_"+str(year)+"_"+ "{month:02d}".format(month=month)+".grib"    
        client.retrieve(dataset, request, target)

CDS API Request for Temperature data from the ERA5 hourly dataset on pressure levels ('renalysis-era5-pressure-levels') on 11-10-2018, from 08:00 to 18:00 UTC on pressure level 1000 hPa.

import cdsapi

dataset = "reanalysis-era5-pressure-levels"
request = {
    "product_type": ["reanalysis"],
    "variable": ["temperature"],
    "year": ["2018"],
    "month": ["10"],
    "day": ["11"],
    "time": [
        "08:00", "09:00", "10:00",
        "11:00", "12:00", "13:00",
        "14:00", "15:00", "16:00",
        "17:00", "18:00"
    ],
    "pressure_level": ["1000"],
    "data_format": "grib",
    "download_format": "unarchived"
}

client = cdsapi.Client()
client.retrieve(dataset, request).download()

CDS API  request for Temperature data from ERA5 complete dataset ('reanalysis-era5-complete') for 01-01-2013 on all model levels (1-137) on 1 degree grid for a specific area

import cdsapi
   c = cdsapi.Client()
   c.retrieve('reanalysis-era5-complete', { # Requests follow MARS syntax
                                            # Keywords 'expver' and 'class' can be dropped. They are obsolete
                                            # since their values are imposed by 'reanalysis-era5-complete'
       'date'    : '2013-01-01',            # The hyphens can be omitted
       'levelist': '1/10/100/137',          # 1 is top level, 137 the lowest model level in ERA5. Use '/' to separate values.
       'levtype' : 'ml',
       'param'   : '130',                   # Full information at https://apps.ecmwf.int/codes/grib/param-db/
                                            # The native representation for temperature is spherical harmonics
       'stream'  : 'oper',                  # Denotes ERA5. Ensemble members are selected by 'enda'
       'time'    : '00/to/23/by/6',         # You can drop :00:00 and use MARS short-hand notation, instead of '00/06/12/18'
       'type'    : 'an',
       'area'    : '80/-50/-25/0',          # North, West, South, East. Default: global
       'grid'    : '1.0/1.0',               # Latitude/longitude. Default: spherical harmonics or reduced Gaussian grid
       'format'  : 'netcdf',                # Output needs to be regular lat-lon, so only works in combination with 'grid'!
   }, 'ERA5-ml-temperature-subarea.nc')     # Output file. Adapt as you wish.

Related articles

• Page:
  Please read: CDS and ADS migrating to new infrastructure: Common Data Store (CDS) Engine
• Page:
  How to migrate to CDS API on the Atmosphere Data Store (ADS)
• Page:
  How to install and use CDS API on Windows
• Page:
  How to install and use CDS API on macOS
• Page:
  How to download ERA5