Versions Compared

Old Version 10

changes.mady.by.user Paul Dando

Saved on

compared with

New Version 11

changes.mady.by.user Unknown User (uscs)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Use MARS list to find out the size

Using You can use the MARS catalogue and the the "View MARS request" functionality  to get a list request of all the data you want can be created which can be used to find out the size and distribution of the raw data in the archive. Note the LIST verb and MARS request and modify the keywords with the information that you need. Once you have the request prepared, you should replace RETRIEVE by LIST and add the OUTPUT = COST keyword. Any post-processing keyword will be ignored by the LIST verb.

Code Block
LIST,
    CLASS      = OD,
    TYPE       = PF,
    STREAM     = ENFO,
    EXPVER     = 0001,
    LEVTYPE    = SFC,
    PARAM      = 134.128/151.128/165.128/166.128/246.228/247.228,
    DATE       = 20170301/to/20170331,
    TIME       = 0000,
    STEP       = 0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/93/96/99/102/105/108/111/114/117/120/123/126/129/132/135/138/141/144/150/156/162/168/174/180/186/192/198/204/210/216/222/228/234/240/246/252/258/264/270/276/282/288/294/300/306/312/318/324/330/336/342/348/354/360,
    NUMBER     = 1/to/50,
    OUTPUT     = cost,
    TARGET     = list.txt

...

Code Block
size=4484574297000; # 4.48 TB
number_of_fields=1348500;
online_size=799241928660;
off_line_size=3685332368340;
number_of_tape_files=93;
number_of_disk_files=281;
number_of_online_fields=240330;
number_of_offline_fields=1108170;
number_of_tapes=32;

Information to interpret the output:

  • The size is in bytes
  • online_size is the number of bytes that are in the MARS disk
  • offline_size is the number of bytes that are in the MARS tapes
  • number_of_tapes tells how the off_line data is split in tapes at this particular time. This may change in the future if for some reason there is more data in the MARS disk
Warning

The data is too large: 4.48 TB and split across 1,348,500 fields, 32 tapes.

Use the MARS catalogue to find out how the data are distributed in files and tapes

...

For this particular case a different "Parameter", "Number" and "Step" can be selected for a specific date and time. All the fields you can choose in this page are stored on the same tape file.

Tip

You should get as much data as possible from this page in a single MARS retrieval request.

Warning

You should not iterate through any of this keywords.

You can use the "Estimated download size" feature or you can run a If we run the list of the sub section of data that we you need and appears appearing in the final stage page we get:

Code Block
LIST,
    CLASS      = OD,
    TYPE       = PF,
    STREAM     = ENFO,
    EXPVER     = 0001,
    LEVTYPE    = SFC,
    PARAM      = 134.128/151.128/165.128/166.128/246.228/247.228,
    DATE       = 20170301,
    TIME       = 0000,
    STEP       = 0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/93/96/99/102/105/108/111/114/117/120/123/126/129/132/135/138/141/144/150/156/162/168/174/180/186/192/198/204/210/216/222/228/234/240/246/252/258/264/270/276/282/288/294/300/306/312/318/324/330/336/342/348/354/360,
    NUMBER     = 1/to/50,
    OUTPUT     = cost,
    TARGET     = list2.txt

...

Tip

In this case the raw data (without post-processing) is ~144GB, 43,500 fields. The total size of the data to be transferred to your system will be less if you interpolate to lat/lon and/or filter the area.

The data is stored across 3 different tapes, which is sensible.

...

The browser can now be used to find out how the data are distributed in the MARS tree. Focussing on If you have a look to the "Current selection" section for this specific example shows:

...

Code Block
languagebash
#!/bin/bash

#this example will filter the area of Europe (N/W/S/E) and interpolate the final fields to a lat/lon 0.5/0.5 degrees
AREA="73.5/-27/33/45"
GRID="0.5/0.5"
 
# fixed selection from the same block
STEP="0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/93/96/99/102/105/108/111/114/117/120/123/126/129/132/135/138/141/144/150/156/162/168/174/180/186/192/198/204/210/216/222/228/234/240/246/252/258/264/270/276/282/288/294/300/306/312/318/324/330/336/342/348/354/360,"
PARAMS="134.128/151.128/165.128/166.128/246.228/247.228"
NUMBER="1/to/50"
 
TIMES="0000"
YEAR="2017"
MONTH="03"

#date loop
for y in ${YEAR}; do

  for m in ${MONTH}; do
    #get the number of days for this particular month/year
    days_per_month=$(cal ${m} ${y} | awk 'NF {DAYS = $NF}; END {print DAYS}')
    
    #date loop
    for my_date in $(seq -w 1 ${days_per_month}); do
      my_date=${YEAR}${MONTH}${my_date}
      
      #time lop
      for my_time in ${TIMES}; do
        cat << EOF > my_request_${my_date}_${my_time}.mars
RETRIEVE,
    CLASS      = OD,
    TYPE       = PF,
    STREAM     = ENFO,
    EXPVER     = 0001,
    LEVTYPE    = SFC,
    GRID       = ${GRID},
    AREA       = ${AREA},
    PARAM      = ${PARAMS},
    DATE       = ${my_date},
    TIME       = ${my_time},
    STEP       = 0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/93/96/99/102/105/108/111/114/117/120/123/126/129/132/135/138/141/144/150/156/162/168/174/180/186/192/198/204/210/216/222/228/234/240/246/252/258/264/270/276/282/288/294/300/306/312/318/324/330/336/342/348/354/360${STEP},
    NUMBER     = 1/to/50${NUMBER},
    TARGET     = enfo_pf_${my_date}_${my_time}.grib
EOF
      mars my_request_${my_date}_${my_time}.mars
      if [ $? -eq 0 ]; then
        rm -f my_request_${my_date}_${my_time}.mars
      fi
      done
    done
  done
done

 

...

Note

You can use the multi-target feature in the TARGET keyword if you are using the full MARS client (from within ECMWF computing facilities).

You can also use ecCodes to post-process the TARGET file afterwards.