Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Confirmed.

Table of Contents

UNDER CONSTRUCTION

What is the objective of this page?

Info

To help users to improve ERA interim MARS requests performance via the WebAPI.

(lightbulb) A good understanding of the MARS efficiency issues is essential especially for users that are interested in downloading large amounts of data.

How is the

...

TIGGE data organised in MARS?

Info

In general it is organised, as a huge tree, with the indentation below, showing different levels down that tree:

  • type of data (analysis, forecast, 4D analysis increments)level 
    •  date
      •  time
        • steps, origins (ie Centres), members
    • year
      • month
        • type of level (model level, pressure level, surface) dates, times, steps, levels, parameters (same tape file (star))

...

        •   

Best practise to iterate over all dates and all times for for a particular level type (eg Pressure levels)

Info

(lightbulb) The idea is to request as much data as possible from the same tape file (star). The natural way to group requests would be:
all parameters, all levels, all time-steps, all dates of a month

(warning) Note the following:

  1. 'all' means 'all' that the user wants. It doesn't have to be all parameters.
  2. If a user is interested only on z500,  he may request more months in one go, since the overall request will not be so big.

An ERA interim request

 

  The best approach is to iterate over the dates and for each date iterate over the times. At this point you may wish to  check TIGGE availability.

Info
for date in dates
for time in times
TIGGE-request(date, time, origin)
(Here you must add everything you need)

Web-API examples:

Example 1: Two TIGGE requests (sfc and pl),  for some dates,  for the same 'origin'

Info
  • The objective of this example is to demonstrate how to iterate efficiently over some years, for a particular ERA interim request. You may wish to have a look on the  ERA interim availabilitydates (eg '2014-10-01' and '2014-11-01')  two requests  for 'perturbed forecast', 'pressure' and 'sfc' level, for TIGGE, ECMWF Center.
  • It The request below can be used as a starting point,however you need to   please keep in mind that you have to adapt it to your needseg to set the keyword, values according to your requirements (eg add or remove "param", "levtype", "step" 'param', 'levtype', 'step' etc).
  • In this way you can extend this example to download the whole ERA interim dataset. these requests to download more data or a longer period of TIGGE.

(thumbs up) The most efficient way to organise your requests is per dates and times

ie select a set of 'dates' and 'times' and add  the 'steps', 'origins', 'levels' and 'parameters' that you need.

(thumbs down)   It is not efficient to organise your requests (smile) You can use the variable target to organise the requested data in files as you wish. per parameter, per  origin  or per step. (eg to submit a different request per parameter or per origin for a list of dates)

Code Block
languagepy
#!/usr/bin/env python
import calendar
from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()
   
def retrieve_tigge_interimdata():
    """       
       A function to demonstrate how to iterate efficiently over all days and months,
dates = ['2014-10-01', '2014-11-01']
    times = ['00', '12']
    for date in dates:
         for time in times:
             target = 'ecmwf_sfc_%s_%s.grb' % (date, time)
             tigge_pf_sfc_request(date, time, target)
             target = 'ecmwf_pl_%s_%s.grb' % (date, time)
        for years 2014 and 2015 for a particular interim_request. 
 tigge_pf_pl_request(date, time, target)

def tigge_pf_pl_request(date, time, target):
    '''
       A TIGGE request for perturbed forecast, pressure level, ECMWF Center.
       Please note Youthat a cansubset extendof the number of years available data is requested below.
       Change the keywords below to adapt theit iteration to your needs. (ie to add more parameters, or numbers etc)
    '''
    server.retrieve({
        'class': 'ti',
  You can also use the variable 'targetdataset': to'tigge',
 organise the requested data in files as you wish.'date': date,
       In the example below the data are organised in files per day. (eg '2015-12-01.grb').
    """'expver': 'prod',
        'grid': '0.5/0.5',
        'levelist': '200/250/300/500/',
        'levtype': 'pl',
    yearStart = 2014    'number': '1/TO/10',
    yearEnd = 2015
    'origin': 'ecmf',
     monthStart = 1   'param': '130/131/132/133',
    monthEnd = 12
    'step': '0/TO/96/BY/12',
        'target': target,
        'time': time,
     for  year in list(range(yearStart, yearEnd + 1)):
'type': 'pf',
    })

def tigge_pf_sfc_request(date, time, target):
    '''
       A TIGGE request for month in list(range(monthStart, monthEnd + 1)):
 perturbed forecast, sfc, ECMWF Center.
       Please note that a subset of the available data is requested below.
       Change the keywords below to adapt it to your needs. (ie to add more parameters, or numbers etc)
    '''
    server.retrieve({
         'class': 'ti',
         'dataset': 'tigge',
         'date': date,
         'expver': 'prod',
    numberOfDays = calendar.monthrange(year, month)[1]
     'grid': '0.5/0.5',
         'levtype': 'sfc',
         'number': '1/TO/10',
         'origin': 'ecmf',
         'param': '146/147/151/165/166/167/168/176/177',
         'step': '0/TO/96/BY/12',
         'target': target,
      for day in list(range(numberOfDays)):
 'time': time,
         'type': 'pf',
        requestDate})

if __name__ == '%04d%02d%02d' % (__main__':
    retrieve_tigge_data()
                                          

Example 2: A TIGGE sfc request for some dates, for several 'origins'

Info

(lightbulb) If you wish to download the same data for more than one origins it is more efficient to request all of them in one go.

  • The objective of this example is to demonstrate how you can get the same data for several origins in one request.
  • Before you start writing your request, you need to check that the requested data are available on all the selected 'origins' .
Code Block
languagepy
#!/usr/bin/env python
from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()
   
def retrieve_tigge_data():
    dates = ['2014-12-01', '2014-12-02']
    times = ['00', '12']
    for date in dates:
         for time in times:
year, month, day + 1)
                target = "'tigge_cf_sfc_%s_%s.grb"' % (requestDatedate, time)
                interimtigge_cf_sfc_request(requestDatedate, time, target)
         

def interimtigge_cf_sfc_request(requestDatedate, time, target):
    '''
   """       
    A TIGGE request for control forecast, sfc, for 3 origins : ECMWF, JMA and KMA.
       Keep  An ERA interim request for analysis, pressure level datain mind that if you wish to download the same data, for more than one origins, 
       it is more efficient to request all of them in one go.
        You can change the keywords below to adapt it to your needs., 
       (ie (egto add more parameters, or removesteps, or levels,even parameters,more timesorigins etc), 
    """   Presumably you need to check the availability of the requested origins.
    '''
    server.retrieve({
        " 'class"': "ei"'ti',
        "stream": "oper",
 'dataset': 'tigge',
         "type"'date': "an"date,
        "dataset": "interim",
 'expver': 'prod',
         "date": requestDate'grid': '0.5/0.5',
        "expver": "1" 'levtype': 'sfc',
        "levtype": "pl",
 'origin': 'ecmf/rjtd/rksl',
         "levelist"'param': "100/200/225/250/300/350/400/450/500/700/750/850/925/1000"'167/168',
        "param" 'step': "129.128/130.128/131.128/132.128"'0/96/168',
        " 'target"': target,
         "'time"': "00/06/12/18"time,
         "grid": "0.75/0.75"
'type': 'cf',
       })

if __name__ == '__main__':
    retrieve_interim()tigge_data()
                                          
Info