Page History

...

ssh ecs-login
cd $SCRATCH

On login node one can find ECS/HPC "mars" command which is installed as part of MARS client and can be executed called without loading any module. The solutions below provide valid MARS requests which can be passed to the mars command as:

...

ls t.grb

This morning you have learned how to use ecCodes. You can use it now to check content of your GRIB file:

...

How many messages the GRIB file contains? Does it match to the number of expected fields retrieved from MARS?

2)

We run the firs MARS request on a login node of ECS. Because of the limitations imposed on interactive sessions (e.g., memory, cpu time) it is better to use batch jobs to run MARS request. Yesterday, you learned how to run fractional jobs. Let's use it know to execute next MARS request. The following template script can be used for this purpose:

#!/bin/bash
  
#SBATCH --job-name=mars_job
#SBATCH --qos=ef
#SBATCH --time=1:00:00
#SBATCH --output=mars.%j.out
#SBATCH --error=mars.%j.out

set -xe

# Enable multi target option!
export MARS_MULTITARGET_STRICT_FORMAT=1 

# to run in debug mode
#export MARS_DEBUG=1

export SCRATCH
export PERM

mars <<EOF
#{paste your MARS request here}
EOF

Use the template above to retrieve wind components valid for 5 days ago at 12:00 and 18:00 from forecast made 7 days ago at 12 UTC at all 137 model levels  > see in MARS Catalogue

retrieve,
    class    = od,
    stream   = oper,
    type     = forecast,
    expver   = 1,
    param    = 131/132, 
    levtype  = model level,
    levelist = 1/to/137,
    date     = -7,
    time     = 12,
    step     = 48/54, 
    target   = "$SCRATCH/wind.grb"

Submit your MARS job with:

ecsbatch mars_job.sh

You can check the job status with:

watch ecsqueue -u $USER

once the MARS job is complete, please check the job's output file, read MARS messages, check for any ERROR or WARNING messages. Which step has taken most of the request time:

From now on, we will use mars_job.sh template to run all MARS requests are batch jobs.

3)

Use the same batch job template (mars_job.sh) from example 2) to run MARS request retrieving the 1000 hPa geopotential from two weeks old 12 UTC forecast (> see in MARS Catalogue) steps: 12 hour to 240 hour with 12 hour intervals:

2)

We run the first MARS request on a login node of ECS. Because of the limitations imposed on interactive sessions (e.g., memory, cpu time) it is better to use batch jobs to run MARS request which can be large with long runtime. Yesterday, you learned how to run fractional jobs. Let's use it know to execute next MARS request. The following template script can be used for this purpose:

#!/bin/bash
  
#SBATCH --job-name=mars_job
#SBATCH --qos=ef
#SBATCH --time=1:00:00
#SBATCH --output=mars.%j.out
#SBATCH --error=mars.%j.out

# For larger requests downloading data from tapes
# extend walltime "--time" to 1 day or longer 

set -xe

# Enable multi target option:
export MARS_MULTITARGET_STRICT_FORMAT=1 

export SCRATCH
export PERM

mars <<EOF
#{paste your MARS request here}
EOF

More examples can be found on: HPC2020: example Slurm serial batch job scripts for ECS

Use the template above to retrieve wind components valid for 5 days ago at 12:00 and 18:00 from forecast made 7 days ago at 12 UTC at all 137 model levels  > see in MARS Catalogue

retrieve,
    class    = od,
    stream   = oper,
    type     = forecast,
    expver   = 1,
    param    = 131/132, 
    levtype  = model level,
    levelist = 1/to/137,
    date     = -7,
    time     = 12,
    step     = 48/54, 
    target   = "$SCRATCH/wind.grb"

Submit your MARS job with:

ecsbatch mars_job.sh

You can check the job status with:

watch ecsqueue -u $USER

once the MARS job is complete, please check the job's output file, read MARS messages, check for any ERROR or WARNING messages. Read "INFO" messages, which action has taken longest time:

From now on, we will use mars_job.sh template to run all MARS requests as batch jobs.

3)

Use the same batch job template (mars_job.sh) from example 2) to run MARS request retrieving the 1000 hPa geopotential from two weeks old 12 UTC forecast (> see in MARS Catalogue) steps: 12 hour to 240 hour with 12 hour intervals:

retrieve,
    class = od,
    stream = oper,
    type = fc,
    expver = 1,
    date = -14,
    time = 12,
    step = 12/to/240/by/12,
    levtype = pl,
    level = 1000,
    param = 129.128,
    target = "$SCRATCH/z1000_c.grb"

4) Fix errors in request:

retrieve,
    class = od,
    date = -7,
    expver = 1,
    levelist = 500/700/1000,
    levtype = pl,
    param = 130.128,
    step = 83/to/93/by/1,
    stream = oper,
    time = 12:00:00,
    type = fc,
    target = "$SCRATCH/temp.pl.grb"

This request returns some errors:

mars - ERROR  - 20231030.070854 - Last error is -43
mars - ERROR  - 20231030.070854 - Expected 33, got 27.
mars - ERROR  - 20231030.070854 - Request failed

Retrieved number of fields is lower than requested. Use MARS catalogue to see which of requested values are not available in MARS. You can use eCcodes to list content of resulting GRIB file and compare it to request:

grib_ls temp.pl.grb

Fix the MARS request by removing non-existing values from the request and re-run it again.

retrieve,
    class = od,
    date = -7,
    expver = 1,
    levelist = 500/700/1000,
    levtype = pl,
    param = 130.128,
    step = 83/84/85/86/87/88/89/90/93,
    stream = oper,
    time = 12:00:00,
    type = fc,
    target = "$SCRATCH/temp.pl.grb"

Post-processing

1)

Retrieve the 1000, 500 and 300 hPa temperatures valid for day before yesterday at noon from the 12 UTC forecast of four days ago on an N48 Gaussian Grid  > see in MARS Catalogue

...

retrieve,
    class = od,
    stream = oper,
    expver = 1, 
    type = forecast,
    param = 130.128,
    date = -4,
    time = 12,
    step = 48,
    grid = 1.5/1.5,
    levelist = 1000/500/300,
    levtype = pl,
    target = "$SCRATCH/t.ll.grb"

Retrieve Europe in European domain on the same way latitude-longlitude grid as in the previous retrieval. Compare the size of the resulting file with the previous one. > see in MARS Catalogue

...

mars - INFO   - ............... - The efficiency of your requests in the last 12 hours is

What does it say, what is efficiency of your MARS usage?

...

 - ............... - The efficiency of your requests in the last 12 hours is

What does it say, what is efficiency of your MARS usage?

This score is based on number of times you access the same data tape during 12 hours. If you access the same tape more than once, the score will be lower than 100%. More accesses to the same tame - lower the score is. Conclusion: One should download all required data from a single tape file in one go to avoid queueing for the same tame multiple times!

Question: You need to download selection of surface parameters from operational analysis at 12 UTC, every day for two consecutive months September and October 2021. How many requests are you going to run?

Tip: Use MARS catalogue to see which analysis are stored together in a tape file.

List 

Find the request size of for control forecast for a historical date without downloading the data:

...

list,
    output=cost,
    class=od,
    date=2020-01-01/to/2020-01-31,
    expver=1,
    levtype=sfc,
    param=167.128,
    step=1/to/96/by/1,
    stream=enfo,
    time=12,
    type=cf

If you need to download this data for entire January 2020, how would you proceed:

• Running one request for entire month (pros / cons)
• Running 31 daily requests (pros / cons)

?

Read, Write, Compute

Retrieve U and V wind components at 10 metre from 10 day old analysis and calculate wind speed > see in MARS catalogue 

...