Identify the data you want to retrieve

Use MARS "list" verb to find out the size of your request

You can use the MARS catalogue and the "View MARS request" to get a MARS request and modify the keywords with the information that you need. Once you have the request prepared, you should replace RETRIEVE by LIST and add the OUTPUT = COST keyword. Any post-processing keyword will be ignored by the LIST verb.

LIST,
    OUTPUT     = COST,
    CLASS      = OD,
    TYPE       = FC,
    STREAM     = OPER,
    EXPVER     = 0001,
    LEVTYPE    = ML,
    LEVELIST   = 127/128/129/130/131/132/133/134/135/136/137,
    PARAM      = 130/131/132,
    DATE       = 20170401/20170402/20170403/20170404/20170405/20170406/20170407/20170408/20170409/20170410/20170411/20170412/20170413/20170414/20170415/20170416/20170417/20170418/20170419/20170420/20170421/20170422/20170423/20170424/20170425/20170426/20170427/20170428/20170429/20170430,
    TIME       = 0000/1200,
    STEP       = 0/to/90/by/1,
    TARGET     = list.txt

When we run the list verb, the following output is written into the file specified with the TARGET keyword (in this example, 'list.txt').

size=591270399720;
number_of_fields=180180;
online_size=206846193282;
off_line_size=384424206438;
number_of_tape_files=60;
number_of_disk_files=546;
number_of_online_fields=63033;
number_of_offline_fields=117147;
number_of_tapes=8;

Information on how to interpret this output:

This request is too large: 180,000 fields, 8 tapes.

Use the MARS catalogue to find out how the data are distributed in files and tapes

Using the MARS catalogue, browse to the data you want to retrieve until you reach the final stage which gives a selection and several options: http://apps.ecmwf.int/mars-catalogue/?class=od&stream=oper&expver=1&type=fc&year=2017&month=apr&levtype=ml&date=2017-04-01&time=00:00:00

For this particular case, a different "Parameter", "Level" and "Step" can be selected for a specific date and time. All the fields you can choose in this page should be stored on the same tape file.

You should get as much data as possible from this page in a single MARS retrieval request.


You should not iterate through any of these keywords.

You can use the "Estimated download size" feature or you can run with the list verb of the sub section of data that you need appearing in this final stage page:

LIST,
    CLASS      = OD,
    TYPE       = FC,
    STREAM     = OPER,
    EXPVER     = 0001,
    LEVTYPE    = ML,
    LEVELIST   = 127/128/129/130/131/132/133/134/135/136/137,
    PARAM      = 130/131/132,
    DATE       = 20170401,
    TIME       = 0000,
    STEP       = 0/to/90/by/1,
    OUTPUT     = cost,
    TARGET     = list2.txt


size=9854506662;
number_of_fields=3003;
online_size=6211981722;
off_line_size=3642524940;
number_of_tape_files=1;
number_of_disk_files=12;
number_of_online_fields=1893;
number_of_offline_fields=1110;
number_of_tapes=1;


In this case the raw data (without post-processing) is ~9.85GB.  The total size of the data to be transferred to your system will be less if you interpolate to lat/lon and/or filter the area.

The data is stored on one single tape, which is sensible and an efficient way to retrieve it.


The size of the file that will be transferred to the system can be established by downloading a single field using the Post-processing keywords (usually GRID and AREA). Then multiply the size of the file obtained containing this single field by the total number of fields that you want to retrieve in a single request. In this case, number_of_fields=3003.

Now that we know the size of the data that we want to retrieve in one go from the same block, we can study the best way to iterate to get the data you need.

Split the request in sensible chunks iterating through the correct keywords

The browser can now be used to find out how the data are distributed in the MARS tree. If you have a look to on the "Current selection" section for this specific example shows:

From the top to the bottom we have to start iterating from the inner loop to the outer loop "time", "date", etc.

This is an example BASH script to loop retrieving data for one date and time at a time for a full month.

Bash script example

#!/bin/bash

# this example will filter the area of Europe (N/W/S/E) and interpolate the final fields to
# a 0.5x0.5 regular lat-lon grid (GRID=0.5/0.5)
AREA="73.5/-27/33/45"
GRID="0.5/0.5"
 
# fixed selection from the same block
PARAMS="130/131/132"
LEVELIST="127/128/129/130/131/132/133/134/135/136/137"
STEP="0/to/90/by/1"
 
TIMES="0000 1200"
YEAR="2017"
MONTH="04"

#date loop
for y in ${YEAR}; do

  for m in ${MONTH}; do
    #get the number of days for this particular month/year
    days_per_month=$(cal ${m} ${y} | awk 'NF {DAYS = $NF}; END {print DAYS}')

    for my_date in $(seq -w 1 ${days_per_month}); do
      my_date=${YEAR}${m}${my_date}

      #time lop
      for my_time in ${TIMES}; do
        cat << EOF > my_request_${my_date}_${my_time}.mars
RETRIEVE,
    CLASS      = OD,
    TYPE       = FC,
    STREAM     = OPER,
    EXPVER     = 0001,
    LEVTYPE    = ML,
    GRID       = ${GRID},
    AREA       = ${AREA},
    LEVELIST   = ${LEVELIST},
    PARAM      = ${PARAMS},
    DATE       = ${my_date},
    TIME       = ${my_time},
    STEP       = ${STEP},
    TARGET     = "oper_ml_${my_date}_${my_time}.grib"
EOF
      mars my_request_${my_date}_${my_time}.mars
      if [ $? -eq 0 ]; then
        rm -f my_request_${my_date}_${my_time}.mars
      fi
      done
    done
  done
done


You can use the multi-target feature in the TARGET keyword if you are using the full MARS client (from within ECMWF computing facilities).

You can also use ecCodes to post-process the TARGET file afterwards.