...

• SFC, 6 parameters
• Time: 00,
• Dates: 1 to 31 March 2017
• all steps (0/to/90/by/1, 93/to/144/by/3 and 150/to/360/by/6)
• all numbers perturbed members (1/to/50)
• Area: Europe
• Output grid: regular lat/lon 0.5/0.5

...

Using the MARS catalogue and the "View MARS request" functionality you can create a list request of all the data you want which would tell you can be created which can be used to find out the size and distribution of the raw data in the archive. Note the LIST verb and the OUTPUT = COST keyword.

...

When we run the list action this is the output, the following output is written to the file specified with the TARGET keyword (in this example, 'list.txt'):

size=4484574297000; # 4.48 TB
number_of_fields=1348500;
online_size=799241928660;
off_line_size=3685332368340;
number_of_tape_files=93;
number_of_disk_files=281;
number_of_online_fields=240330;
number_of_offline_fields=1108170;
number_of_tapes=32;

Use the MARS catalogue to

...

find out how the data

...

are distributed in files and tapes

Using the MARS catalogue we , browse to the data we you want to get until we get to retrieve until you reach the final stage which gives us a selection and several options:   http://apps.ecmwf.int/mars-catalogue/?stream=enfo&levtype=sfc&time=00%3A00%3A00&expver=1&month=mar&year=2017&date=2017-03-01&type=pf&class=od

For this particular case we can choose a different "Parameter", "Number" and "Step" . This means that all can be selected for a specific date and time. All the fields you can choose in this page are in stored on the same tape file. Therefore users You should get as much data as possible from this page in a single MARS retrieval request.

If we run the list of the data that we need and appears in the final stage page we get:

...

Now that we know the size of the data that we want to retrive retrieve in one go from the same hypercube we can start the study of the way how best to iterate to get retrieve the full month of data required.

Split the request in sensible chunks iterating through the correct keywords

Using the browser we know The browser can now be used to find out how the data is are distributed in the MARS tree. Now we focus Focussing on the current "Current selection" section . In this casefor this specific example shows:

From the top to the bottom we have to start iterating from the inner loop to the outer loop "Time", "date", .etc..

This is an example BASH script to loop 1 time/ 1 day in one go for one month, retrieving data for one date and time at a time for a full month.

#!/bin/bash

#this example will filter the area of Europe (N/W/S/E) and interpolate the final fields to a lat/lon 0.5/0.5 degrees
AREA="73.5/-27/33/45"
GRID="0.5/0.5"
PARAMS="134.128/151.128/165.128/166.128/246.228/247.228"
TIMES="0000"
YEAR="2017"
MONTH="03"

#date loop
for y in ${YEAR}; do

  for m in ${MONTH}; do
    #get the number of days for this particular month/year
    days_per_month=$(cal ${m} ${y} | awk 'NF {DAYS = $NF}; END {print DAYS}')

    for my_date in $(seq -w 1 ${days_per_month}); do
      my_date=${YEAR}${MONTH}${my_date}
      
      #time lop
      for my_time in ${TIMES}; do
        cat << EOF > my_request_${my_date}_${my_time}.mars
REQUESTRETRIEVE,
    CLASS      = OD,
    TYPE       = PF,
    STREAM     = ENFO,
    EXPVER     = 0001,
    LEVTYPE    = SFC,
    GRID       = ${GRID},
    AREA       = ${AREA},
    PARAM      = ${PARAMS},
    DATE       = ${my_date},
    TIME       = ${my_time},
    STEP       = 0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/93/96/99/102/105/108/111/114/117/120/123/126/129/132/135/138/141/144/150/156/162/168/174/180/186/192/198/204/210/216/222/228/234/240/246/252/258/264/270/276/282/288/294/300/306/312/318/324/330/336/342/348/354/360,
    NUMBER     = 1/to/50,
    TARGET     = enfo_pf_${my_date}_${my_time}.grib
EOF
      mars my_request_${my_date}_${my_time}.mars
      if [ $? -eq 0 ]; then
        rm -f my_request_${my_date}_${my_time}.mars
      fi
      done
    done
  done
done      

...