Table of Contents

Identify the data you want

SFC, 6 parameters
Time: 00, March 2017
Dates: 1 to 31 March 2019
all steps (0/to/90/by/1, 93/to/144/by/3 and 150/to/360/by/6)
all numbers perturbed members (1/to/50)
Area: Europe
Output grid: regular lat/lon 0.5/0.5

Use MARS list to find out the size

You can use the MARS catalogue and the "View MARS request" to get a MARS request and modify the keywords with the information that you need. Once you have the request prepared, you should replace RETRIEVE by LIST and add the OUTPUT = COST keyword. Any post-processing keyword will be ignored by the LIST verb.

Code Block

LIST,
    CLASS      = OD,
    TYPE       = PF,
    STREAM     = ENFO,
    EXPVER     = 0001,
    LEVTYPE    = SFC,
    PARAM      = 134.128/151.128/165.128/166.128/246.228/247.228,
    DATE       = 2017030120190301/to/2017033120190331,
    TIME       = 0000,
    STEP       = 0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/93/96/99/102/105/108/111/114/117/120/123/126/129/132/135/138/141/144/150/156/162/168/174/180/186/192/198/204/210/216/222/228/234/240/246/252/258/264/270/276/282/288/294/300/306/312/318/324/330/336/342/348/354/360,
    NUMBER     = 1/to/50,
    OUTPUT     = cost,
    TARGET     = list.txt

When we run the list action this is the output, the following output is written to the file specified with the TARGET keyword (in this example, 'list.txt'):

Code Block

size=4484574297000; # 4.48 TB
number_of_fields=1348500;
online_size=7992419286601491532497000;
off_line_size=36853323683402993041800000;
number_of_tape_files=93;
number_of_disk_files=281495;
number_of_online_fields=240330448500;
number_of_offline_fields=1108170900000;
number_of_tapes=32;19;

Information to interpret the output:

The size is in bytes
online_size is the number of bytes that are in the MARS disk
offline_size is the number of bytes that are in the MARS tapes
number_of_tapes tells how the off_line data is split in tapes at this particular time. This may change in the future if for some reason there is more data in the MARS disk

Warning
The data is too large: 4.48 TB and split across 32 1,348,500 fields, 19 tapes.

Use the MARS catalogue to

...

find out how the data

...

are distributed in files and tapes

Using the MARS catalogue we , browse to the data we you want to get until we get to retrieve until you reach the final stage which gives us a selection and several options: http://apps.ecmwf.int/mars-catalogue/?stream=enfo&levtype=sfc&time=00%3A00%3A00&expver=1&month=mar&year=20172019&date=20172019-03-01&type=pf&class=od

Image Added

For this particular case we can choose a different "Parameter", "Number" and "Step" . This means that all can be selected for a specific date and time. All the fields you can choose in this page are in should be stored on the same tape file. Therefore users

Tip
You should get as much data as possible from this page in a single MARS retrieval request.

Warning
You should not iterate through any of these keywords.

You can use the "Estimated download size" feature or you can run a list of the sub section of data that you need appearing .If we run the list of the data that we need and appears in the final stage page we get:

Code Block

LIST,
    CLASS      = OD,
    TYPE       = PF,
    STREAM     = ENFO,
    EXPVER     = 0001,
    LEVTYPE    = SFC,
    PARAM      = 134.128/151.128/165.128/166.128/246.228/247.228,
    DATE       = 2017030120190301,
    TIME       = 0000,
    STEP       = 0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/93/96/99/102/105/108/111/114/117/120/123/126/129/132/135/138/141/144/150/156/162/168/174/180/186/192/198/204/210/216/222/228/234/240/246/252/258/264/270/276/282/288/294/300/306/312/318/324/330/336/342/348/354/360,
    NUMBER     = 1/to/50,
    OUTPUT     = cost,
    TARGET     = list2.txt

...

Code Block

size=144663687000;
number_of_fields=43500;
online_size=4323282600062355037500;
off_line_size=10143086100082308649500;
number_of_tape_files=3;
number_of_disk_files=1014;
number_of_online_fields=1300018750;
number_of_offline_fields=3050024750;
number_of_tapes=31;

Tip

In this case the raw data (without post-processing) is ~144GB, which would be reduced 43,500 fields. The total size of the data to be transferred to your system will be less if you interpolate to lat/lon and/or filter the area.

The data is split in 3 different tapes, which is sensiblestored in 1 tape.

Tip
The size of the file that will be transferred to the system can be established by downloading a single field using the Post-processing keywords (usually GRID and AREA). Then multiply the size of the file obtained containing this single field by the total number of fields that you want to retrieve in a single request: number_of_fields=43500.

Now that we know the size of the data that we want to retrive retrieve in one go from the same hypercube block we can start the study of the way how best to iterate to get retrieve the full month of data required.

Split the request in sensible chunks iterating through the correct keywords

Using the browser we know The browser can now be used to find out how the data is are distributed in the MARS tree. Now we focus on the current selection section. In this caseIf you have a look to the "Current selection" section for this specific example shows:

From the top to the bottom we have to start iterating from the inner loop to the outer loop "Time", "date", etc...

This is an example BASH script to loop 1 time/ 1 day in one go for one month, retrieving data for one date and time at a time for a full month.

Bash script example

Code Block

language	bash

#!/bin/bash

#this example will filter the area of Europe (N/W/S/E) and interpolate the final fields to a lat/lon 0.5/0.5 degrees
AREA="73.5/-27/33/45"
GRID="0.5/0.5"
 
# fixed selection from the same block
STEP="0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/93/96/99/102/105/108/111/114/117/120/123/126/129/132/135/138/141/144/150/156/162/168/174/180/186/192/198/204/210/216/222/228/234/240/246/252/258/264/270/276/282/288/294/300/306/312/318/324/330/336/342/348/354/360,"
PARAMS="134.128/151.128/165.128/166.128/246.228/247.228"
NUMBER="1/to/50"
 
TIMES="0000"
YEAR="20172019"
MONTH="03"

#date loop
for y in ${YEAR}; do

  for m in ${MONTH}; do
    #get the number of days for this particular month/year
    days_per_month=$(cal ${m} ${y} | awk 'NF {DAYS = $NF}; END {print DAYS}')
    
    #date loop
    for my_date in $(seq -w 1 ${days_per_month}); do
      my_date=${YEAR}${MONTHm}${my_date}
      
      #time lop
      for my_time in ${TIMES}; do
        cat << EOF > my_request_${my_date}_${my_time}.mars
REQUESTRETRIEVE,
    CLASS      = OD,
    TYPE       = PF,
    STREAM     = ENFO,
    EXPVER     = 0001,
    LEVTYPE    = SFC,
    GRID       = ${GRID},
    AREA       = ${AREA},
    PARAM      = ${PARAMS},
    DATE       = ${my_date},
    TIME       = ${my_time},
    STEP       = 0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/93/96/99/102/105/108/111/114/117/120/123/126/129/132/135/138/141/144/150/156/162/168/174/180/186/192/198/204/210/216/222/228/234/240/246/252/258/264/270/276/282/288/294/300/306/312/318/324/330/336/342/348/354/360${STEP},
    NUMBER     = 1/to/50${NUMBER},
    TARGET     = "enfo_pf_${my_date}_${my_time}.grib"
EOF
      mars my_request_${my_date}_${my_time}.mars
      if [ $? -eq 0 ]; then
        rm -f my_request_${my_date}_${my_time}.mars
      fi
      done
    done
  done
done

...

Note

You can use the multi-target feature in the TARGET keyword if you are using the full MARS client (from within ECMWF computing facilities).

You can also use ecCodes to post-process the TARGET file afterwards.

Space shortcuts

Page tree

Versions Compared

Old Version 7

New Version Current

Key

Identify the data you want

Use MARS list to find out the size

Use the MARS catalogue to

find out how the data

are distributed in files and tapes

Split the request in sensible chunks iterating through the correct keywords

Bash script example

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 7

New Version Current

Key

Identify the data you want

Use MARS list to find out the size

Use the MARS catalogue to

find out how the data

are distributed in files and tapes

Split the request in sensible chunks iterating through the correct keywords

Bash script example