Background
We need to retrieve three types of files efficiently for the two reanalyses with experiment id = 21 and experiment id = 22. These are all class=ea. The three "streams" are oper, moda and mfb:
- Stream: Atmospheric daily analyses (oper)
- Stream: Monthly means of Daily Means (moda)
- Stream: Observation departures for the data used in the reanalyses (mfb)
Retrieval of daily "oper" files
The availability of the oper files for experiment = 21 is given here, and the availability of experiment = 22 is here. Please note that these pages also enable users to check estimate the download costs for the retrievals.
The example below retrieves geopotential and temperature on a set of pressure levels for "rom_saf_21" (experiment id = 21). Its can be modified easily for rom_saf_22 (experiment id = 22). The request is for data from 20140101 to 20151231. For efficiency reasons, the retrieval is done one calendar month at a time. These monthly files can be combined with the "cat" command if required. E.g., cat month1 month2 > both_months.
#!/usr/bin/env python
import calendar
from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()
def retrieve_rom_saf():
"""
A function to demonstrate how to iterate efficiently over several years and months etc
for a particular rom_saf request.
Change the variables below to adapt the iteration to your needs.
You can use the variable 'target' to organise the requested data in files as you wish.
In the example below the data are organised in files per month. (eg "rom_saf_21_daily_201510.nc")
"""
yearStart = 2014
yearEnd = 2015
monthStart = 1
monthEnd = 12
for year in list(range(yearStart, yearEnd + 1)):
for month in list(range(monthStart, monthEnd + 1)):
startDate = '%04d%02d%02d' % (year, month, 1)
numberOfDays = calendar.monthrange(year, month)[1]
lastDate = '%04d%02d%02d' % (year, month, numberOfDays)
target = "rom_saf_21_daily_%04d%02d.nc" % (year, month) # or name rom_saf_22_ ...
requestDates = (startDate + "/TO/" + lastDate)
rom_saf_request(requestDates, target)
def rom_saf_request(requestDates, target):
"""
An ROM SAF request for analysis pressure level data.
Change the keywords below to adapt it to your needs.
(eg to add or to remove levels, parameters, times etc)
Request cost per month is 3472 fields, ~200 Mbytes
"""
server.retrieve({
"class": "ea", # do not change
"dataset": "rom_saf_21", # or "rom_saf_22"
"expver": "21", # or "22"
"stream": "oper", # do not change
"type": "an", # do not change, only the analysis is available
"date": requestDates, # dates, set automatically from above
"levtype": "pl", # pressure level data (versus surface, sfc, and model level, ml)
"levelist": "100/500/700/750/850/925/1000", # levels, required only with levtype:pl and levtype:ml
"param": "129.128/130.128", # Geopotential height, Temperature (T), see http://apps.ecmwf.int/codes/grib/param-db
"target": target, # output file name, set automatically from above
"time": "00/03/06/09/12/15/18/21", # times of analysis (with type:an), or initialization time of forecast (with type:fc)
"grid": "1.5/1.5", # Optional for GRIB, required for NetCDF. The horizontal resolution in decimal degrees. If not set, the archived grid as specified in the data documentation is used.
"format": "netcdf", # Optional. Output in NetCDF format. Requires that you also specify 'grid'. If not set, data is delivered in GRIB format, as archived.
})
if __name__ == '__main__':
retrieve_rom_saf()
Retrieval of "moda" files
!/usr/bin/env python
import calendar
from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()
def retrieve_romsaf_data():
'''
A function to demonstrate how to retrieve rom_saf monthly mean data.
Change the variables below to adapt to your needs.
ROM SAF monthly data is timestamped to the first of the month, hence dates
have to be specified as a listin this format:
'20070201/20070301/20070401/.../20151101/20151201'.
Note that 20070101 is not available
Data is stored on one tape per decade, so for efficiency we split the date range into
decades, hence we get multiple date lists by decade:
'20070201/20070201/20000301/.../20051101/20051201'
In the example below the output data are organised as one file per decade:
'romsaf_moda_2010'
Please note that only decades 2000-2010 are available. The First available moda is 20070201!
'''
yearStart = 2007 # adjust to your requirements - only 2007-02-01 to 2015-12-01 is available
yearEnd = 2015 # adjust to your requirements
months = [1,2,3,4,5,6,7,8,9,10,11,12] # adjust to your requirements
months_2007 = [2,3,4,5,6,7,8,9,10,11,12] # specific for 2007 because 200701 not computed
years = range(yearStart, yearEnd+1)
print 'Years: ',years
decades = list(set([divmod(i, 10)[0] for i in years]))
decades = [x * 10 for x in decades]
decades.sort()
print 'Decades:', decades
# loop through decades and create a month list
for d in decades:
requestDates=''
for y in years:
if ((divmod(y,10)[0])*10) == d:
if y == 2007:
for m in months_2007:
requestDates = requestDates+str(y)+(str(m)).zfill(2)+'01/'
else:
for m in months:
requestDates = requestDates+str(y)+(str(m)).zfill(2)+'01/'
requestDates = requestDates[:-1]
print 'Requesting dates: ', requestDates
target = '/home/st/sti/metview/new_neanlysis_scripts/romsaf_moda_22_%d.grb'% (d) # specifies the output file name
print 'Output file: ', target
romsaf_request(requestDates, d, target)
# the actual data request
def romsaf_request(requestDates, decade, target):
'''
Change the keywords below to adapt to your needs.
The easiest way to do this is:
1. go to https://apps.ecmwf.int/mars-catalogue/?class=ea&stream=moda&expver=22
2. click through to the parameters you want, for any date
3. click 'View the MARS request'
'''
server.retrieve({
'class': 'ea', # do not change
'dataset': 'rom_saf_22', # or rom_saf_21
'expver': '22', # or 21
'stream': 'moda', # monthly means of daily means
'type': 'an', # analysis (versus forecast, fc)
'levtype': 'pl', # pressure data (versus pressure level, pl, and model level, ml)
'levelist': '70', # levels, required only with levtype:pl and levtype:ml
'param': '131', # zonal wind
"grid": "1.5/1.5", # Optional for GRIB, required for NetCDF. The horizontal resolution in decimal degrees. If not set, the archived grid as specified in the data documentation is used.
'date': requestDates, # dates, set automatically from above
'decade': decade, # decade set automatically from above
'target': target # output file name, set automatically from above
})
if __name__ == '__main__':
retrieve_romsaf_data()
Retrieval of "mfb" files
#!/usr/bin/env python
import calendar
from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()
def retrieve_rom_saf_mfb():
"""
A function to demonstrate how to iterate efficiently over several years and months etc
for a particular rom_saf mfb request.
Change the variables below to adapt the iteration to your needs.
You can use the variable 'target' to organise the requested data in files as you wish.
In the example below the data are organised in files per month. (eg "rom_saf_mfb_201410.odb")
Each monthly file is ~1.2 Gb for GPSRO data used exp id = 21.
"""
yearStart = 2014
yearEnd = 2014
monthStart = 1
monthEnd = 12
for year in list(range(yearStart, yearEnd + 1)):
for month in list(range(monthStart, monthEnd + 1)):
startDate = '%04d%02d%02d' % (year, month, 1)
numberOfDays = calendar.monthrange(year, month)[1]
lastDate = '%04d%02d%02d' % (year, month, numberOfDays)
target = "/scratch/rd/sti/mfb_test/rom_saf_mfb_%04d%02d.odb" % (year, month)
requestDates = (startDate + "/TO/" + lastDate)
rom_saf_request(requestDates, target)
def rom_saf_request(requestDates, target):
"""
An ROM SAF request for a month of MFB observation data.
Change the keywords below to adapt it to your needs.
(eg to add or to remove levels, parameters, times etc)
"""
server.retrieve({
"class" : "ea", # do not change
"dataset" : "rom_saf_21", # or "rom_saf_22"
"expver" : "21", # or "22"
"stream" : "oper", # do not change
"type" : "mfb", # do not change - only mfb's available for 21 and 22
"date" : requestDates, # dates, set automatically from above
'obsgroup' : '8', # GPSRO - see https://apps.ecmwf.int/odbgov/group/
"target" : target, # output file name, set automatically from above
"time" : "00/12", # times of analysis (00 or 12 or both)
"format" : "odb", # or ascii format
})
if __name__ == '__main__':
retrieve_rom_saf_mfb()