Atmosphere Data Store (ADS) documentation

Introduction

The Atmosphere Data Store (ADS) is the cornerstone infrastructure which supports the implementation of the Copernicus Atmosphere Monitoring Service (CAMS). It is built on the same infrastructure as the Climate Data Store (CDS). It enables the provision of reliable data and expertise related to air quality, solar energy, and the role atmospheric gases and particles play in climate change.

The ADS :

is designed as a distributed system which provides improved access to local and remote datasets via a powerful service-oriented architecture.
offers seamless web-based and API-based search and retrieve facilities to access air quality data and information.

The data provided by the ADS are free and open data, subject to the user agreeing to the relevant dataset licence(s). For further details about the underlying infrastructure, please see this article from the ECMWF Newsletter 151.

How the ADS works

The ADS provides a Catalogue which lists CAMS data and products, such as observations, reanalyses and air quality forecasts. Users can search through the catalogue, and can filter the entries by means of both a faceted search (Variable domain, Parameter family, Spatial coverage, Product type, Temporal coverage) and a textual search.

Users can request data from the ADS using a variety of methods:

the ADS web interface
the CDS Application Programming Interface (CDS API)

1) The ADS web interface is an interactive system: the user fills a web form to construct their query (please note that only a valid query can be submitted via the web form).

Please make sure that you select all the mandatory fields before submit a request.

The user can then currently choose between two options:

submit the form and download the result of the query;
show the query as an API request

In the future, a third option - the ADS Toolbox - will be added.

These are shown in the following figures:


Figure 1 Web interface	Figure 2 Request options on the web interface

2) The CDS API is a service providing programmatic access in Python to ADS data. CDS API example queries can be conveniently constructed by using the ADS web interface and clicking on the "Show API request" button. For a description and some useful examples, please see How to use the CDS API.

Please note: Currently, all ADS datasets are covered by Copernicus Products Licence, and users have to accept it in order to be able to download datasets (whether through the web interface, or the CDS API). If the licence has not been accepted, the error message below will shown by the CDS API:

At the time this article was written, the acceptance of the licence can only be carried out through the ADS web interface for the relevant dataset, by forming a request on the "Download data" page. The user will then be prompted to accept the licence. Once that is done, the CDS API request can be successfully submitted to the ADS.

As the ADS is a distributed platform, the datasets are hosted at a number of different locations. Some data are stored on the ADS disks themselves; others are hosted remotely on the data providers' storage systems, e.g. the ECMWF Meteorological Archival and Retrieval System (MARS).

Table 2 shows where the ADS datasets are currently stored.

Please note that the items the user selects to include in a single ADS request can make a significant difference in terms of performance i.e. how long a given request will take to run. Please see the page Efficiency tips for how to build an efficient request.

On the ADS cache disks, requests and relevant results are generally stored for 1.5 to 2 days, depending on the user load on the ADS system. After this period, files can be deleted, starting from the oldest.

Please note that cached data (if present) is used to fulfill an ADS request only if the ADS request is exactly identical to a previous one. This means that data will be extracted from the cache, rather than being re-retrieved from the dataset itself.

Request states

After sending a request, the user can track its state on the 'Your Requests' page of the ADS web interface. There are five different states of a request:

Queued. Each request is assigned a unique ID and a priority. The priority is chosen according to different criteria, such as the origin of the request (ADS web interface/API). For example, the ADS web interface usually has higher priority because it is an interactive application and users expect an immediate response to their request.
In progress. The request is being fulfilled and the data is being collected from the relevant datasets.
Failed. The request encountered problems and did not complete.
Unavailable. The data has expired from cache and therefore cannot be retrieved at the current time. In this case the request should be resubmitted by the user.
Complete. The resulting data file is ready to download for ADS web interface requests (Completed CDS API requests are not shown, please see note below).

User can check the live status of the ADS queues here https://ads.atmosphere.copernicus.eu/live/queue.

Users can also check the overall status of the ADS system at the live page.

Please note:

Users can also follow the status of their CDS API request on the 'Your Requests' ADS web page, while the request is actually running.
If the CDS API request completes successfully, the data file will be automatically downloaded to the user's computer. The details of this request will be removed from the 'Your Requests' ADS web page, and the data file will not be downloadable from this page.
If the request fails, the details of this CDS API request will remain on the 'Your Requests' ADS web page, for further investigation.

Limits

Limits are set on usage of ADS resources to ensure an appropriate level of performance for users.

They are divided in three categories: 1) per-user, 2) global and 3) system, and the current values for these limits are shown in Table 1 (last reviewed on 26 Jun 2020).

Please note that these limits are changed from time to time according to the current workload of the system and number of concurrent tasks.

The ADS will queue requests which would otherwise cause any of these limits to be exceeded.

The 'live' status of the limits values can be checked here https://ads.atmosphere.copernicus.eu/live/limits

Per-user	Limit on simultaneous tasks/requests
Any request (web interface and API together)	12
Requests that access the ECMWF archive	1
Requests that access the online ADS data	40
Global
Requests that can access archived regional forecast data	1
Requests that can access latest regional forecast data	1
Requests that access the ECMWF archive	30
Requests that access the online ADS data	180
System
Tasks of type 'adaptor'	36
Tasks of type 'cdscompute'	96

Also, ADS data requests have limits in terms of number of fields and volume size, which are different for each dataset (see the "Datasets" Table 2 below). These values are dependent on each dataset's structure and where the dataset is actually stored. Again, these restrictions are introduced to help the system maintain good performance and minimise the queuing time for all users.

For example, when a user ADS web request exceeds the number of fields limit, an information message appears at the bottom of the web interface page (Figure 3). Please note, that these limits are also enforced for requests sent via the CDS API.

Figure 3 Information message when a user ADS web request exceeds the number of fields limit.

Datasets

Dataset	Number of fields limit	Adaptor (indicates where the data are stored; see "Efficiency Tips" below)	Notes
CAMS global atmospheric composition forecasts	10000	MARS Internal/external	Those variables listed as fast-access are readily available from ADS disks. Those variables listed as slow-access are stored in ECMWF MARS tape archive.
CAMS global reanalysis (EAC4)	100000	MARS Internal/external	Those variables listed as fast-access are readily available from ADS disks. Those variables listed as slow-access are stored in ECMWF MARS tape archive.
CAMS global reanalysis (EAC4) monthly averaged fields	100000	MARS Internal	N/A
CAMS global inversion-optimised greenhouse gas fluxes and concentrations	500	adaptor.url	Stored locally on ADS disks
CAMS solar radiation time-series	10000	adaptor.cams_solar_rad2.retrieve	This data is calculated on demand by a service jointly provided by DLR, Armines, and Transvalor. The ADS forwards requests to this service and returns the data from it.
CAMS European air quality forecasts	5000	adaptor.cams_regional_fc.retrieve	There are three classes of speed at which these requests will be processed: Surface ensemble stored on ADS disks: fast-access Recent forecasts and analyses for other models/levels stored by third party: slow-access All other data are stored in a third party archive system: access to this data is in general much slower than 1. and 2.
CAMS global emission inventories	1000	adaptor.url2.retrieve	N/A
CAMS global radiative forcings	1000	adaptor.url
CAMS global radiative forcing - auxilliary variables	1000	adaptor.url

Efficiency tips

Where the data are actually stored can make a significant difference in the speed at which a request is processed. ADS data hosted in 'MARS internal' is stored on ADS disks, and so is faster to retrieve. The 'MARS external' datasets are stored in the ECMWF MARS (tape) archive, and in this case it is important to request as much data as possible from the same tape file in your ADS request.
It is better to submit small requests rather than very large requests. This will ensure your requests are not given a lower priority in the ADS request queue.
When using the CDS API, it is strongly recommended that users take as a starting point the example API request script shown at the bottom of the ADS web 'Download data' page for the dataset of interest, and use this as the basis for your request.

CDS API examples

CDS API Request for sulphate aerosol optical depth 550nm from CAMS global reanalysis (EAC4) dataset on on 01-01-2004, at 00 UTC.

CDS API request example

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'cams-global-reanalysis-eac4',
    {
        'variable': 'sulphate_aerosol_optical_depth_550nm',
        'date': '2004-01-18/2004-01-18',
        'time': '00:00',
        'format': 'netcdf',
    },
    'download.nc')

This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS).

The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose.

The users thereof use the information at their sole risk and liability. For the avoidance of all doubt, the European Commission and the European Centre for Medium-Range Weather Forecasts have no liability in respect of this document, which is merely representing the author's view.

Space shortcuts

Page tree