...
...
| Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Efficiency tips
Efficiency tips
- Where the data are Where the data are actually stored can make a significant difference in performance. CDS data hosted in 'MARS internal' is stored on CDS disks, and so is faster to retrieve. The 'MARS external' datasets are stored in the ECMWF MARS (tape) archive, and in this case it is important to request as much data as possible from the same tape file in your CDS request.
- Submit small requests over very large and heavy requests. This will ensure your requests are not penalised in the CDS request queue. In particular:
For ERA5 data requests, please see: How to download ERA5. The following example shows how to efficiently download a whole year of hourly data for 2m temperature (grib format, for an area subset) from the CDS, by asking for one month of data per request:
Code Block language py title Request for hourly data for 2m temperature for 2018 and 2019 #!/usr/bin/env python3 import cdsapi c = cdsapi.Client() first_year = 2018 last_year = 2019 for year in range(first_year, last_year + 1): for month in range(1, 13): print("=========================================================") print("Downloading {year}-{month:02d}".format(year=year, month=month)) c.retrieve( 'reanalysis-era5-single-levels', { 'product_type': 'reanalysis', 'variable': '2m_temperature', 'year': str(year), 'month': "{month:02d}".format(month=month), 'day': [ '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', ], 'time': [ '00:00', '01:00', '02:00', '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00', '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00', '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00', ], 'area': [ 90, 170, 80, 180, ], 'format': 'grib', }, "{year}-{month:02d}.grib".format(year=year, month=month))- For C3S seasonal forecast data requests, please see: Recommendations and efficiency tips for C3S seasonal forecast datasets.
- For UERRA data requests, please see: UERRA retrieval efficiency.
- When using the CDS API or the CDS Toolbox, it is advised that users take as an example the API request or toolbox request script shown at the bottom of the CDS web 'Download data' page for the dataset of interest, and use this as the basis for your request.
- Some ERA5 datasets, such as
reanalysis-era5-complete, do not appear in the CDS catalogue on the web interface, but users can still retrieve the data through the CDS API. In these cases, users can make use of a subset of the ECMWF MARS keywords in their CDS API request, and should also follow the MARS efficiency rule of thumb (the idea is to request as much data as possible from the same tape file or to reduce the number of tapes involved.). - Users can check the allowed and not allowed keywords for the CDS API requests in the following article: Climate Data Store (CDS) API Keywords
...