A MARS request is the way to specify an action on a set of fields or observations. The directives specified in a MARS request have the following syntax:
verb, keyword1 = value 1, ... = ..., keywordN = value N |
Where:
- verb identifies the action to be taken, e.g. retrieve or list.
- keyword is a known MARS variable, e.g. type or date
- value is the value assigned to the keyword, e.g. Analysis or temperature
Notes on MARS request syntax:
- Leading spaces and tab characters are ignored.
- verbs and directives keyword=value have to be terminated with a comma, except the last one, which MARS considers as the end of a request.
- Directives may be in upper or lower case or a mixture of both; MARS converts everything except filenames to upper case.
- Characters * ! and # can be used to comment until end-of-line.
- Values may be predefined names, numeric values or user-supplied character strings such as filenames.
- Abbreviations may be used for verbs, keywords and values, provided enough letters are given to uniquely identify what is meant.
- MARS has acronyms for the names of different parameters which can be specified by users.
- Keywords may be assigned a single value, a list of values or a range of values. A list is indicated by the separator / ; a range is indicated by using the keywords to as well as / and by. Examples of different formats for values are given in the table below.
Examples of MARS keyword assignments Format Example single value param = temperature/SSRD list of values step = 12/24/48 range of values date = 19990104/to/19990110/by/2
The number of fields described in a MARS request is computed by multiplying the number of values of its identification keywords after the expansion of all ranges. This number, or the value specified by expect, is used by MARS to decide whether the request has been fully satisfied or to fail otherwise.
Certain keywords accept all as a valid value, e.g. param = all. In this case, MARS will deliver data available matching the rest of the request. Please note that if all is assigned to a keyword, MARS will no longer check on missing values. Furthermore, if all is assigned, MARS can retrieve data from the archive only, not from the Fields Data Bases.
Most keywords accept off as a valid value. In this case, the keyword is not considered by MARS and will be removed from the request before processing it.
The following example shows a retrieve request with directives grouped according to their meaning:
retrieve action class=od stream = oper, identification of archive expver = 1, date = -1, date and time related type = analysis, levtype = model levels, levelist = 1/to/91, param = temperature, grid = 2.5/2.5, post-processing area = europe, use = infrequent, execution control target = analysis.12 storage
Retrieve requests have to specify, at least, directives to identify the data and directives for storage of the retrieved data. The ones to identify fields can be found in the Archive Catalogue. Data manipulation (post-processing) directives and execution control directives are optional, depending on user's needs.
With the main focus on data retrievals, this User Guide covers the following MARS actions, or verbs:
retrieve to extract data from MARS or from a source file. Most of this documentation will cover the usage of the retrieve verb.
compute will perform mathematical operations on retrieved grid point fields in GRIB format, see section 10.
list queries the MARS archive about the data availability for a given request, without actually retrieving the data. It is suitable for interactive or batch use to check availability of certain datasets and its collocation in the archive, see section 9
read can be used to filter or manipulate data already retrieved into a file, see section 5.4.
write is mainly used to save data from fieldsets, a temporary storage, into files, see section 5.2.
Fieldsets
A fieldset is a temporary storage for fields, either retrieved from MARS or read from a UNIX file. Fieldsets can be seen as variables in a MARS request, and they can be referenced in subsequent MARS requests in the same execution. This is an example how to retrieve data into a fieldset named analysis:
retrieve, class = od, type = analysis, stream = oper, expver = 1, date = -1, time = 12, levtype = pressure levels, levelist = 1000/850/500, param = temperature, fieldset = analysis |
At the end of the execution, all fieldsets are released. Therefore, data in unsaved fieldsets will be lost. Fieldsets can be saved as UNIX files with write requests, like in this example:
write, fieldset = analysis, target = "data" |
Note that fieldsets are in fact using temporary disk storage on the host where MARS is executed, usually in $TMPDIR directory. When dealing with big fieldsets, users are advised to make such variable point to a suitable directory were temporary data can be stored (e.g. $SCRATCH on ecgate).
One can extract fields (sub fieldsets) from a fieldset using square brackets. If X is a fieldset, X[2] is the second field, X[1,10] comprises the first 10 fields, and X[2,8,2] comprises every second field of x starting from 2 up to 8.
Multiple requests in one call to MARS
MARS can accept more than one request in the same invocation. This allows users to do further computations than the ones provided by the standard post-processing directives before writing the data into a target file and it is how the compute verb can be used. The following concepts have to be taken into account:
Keyword inheritance
When two or more retrieve requests are used in a single call to MARS, keywords not explicitly set in the second request are inherited from the first (previous). The following example retrieves 12 UTC model level and 1000 hPa pressure level temperature for yesterday's analysis:
retrieve, class = od, type = analysis, stream = oper, date = -1, time = 12, levtype= model levels, level = 1/to/91, param = temperature, target = analysis.ml retrieve, levtype = pl, levelist= 1000, target = analysis.pl |
There can be cases in which unwanted keywords are inherited. They can be removed from subsequent requests by assigning the value off.
Append to target file
It is possible to store data from different requests into the same target file. Note that MARS will append the result of each retrieve request at the end of the target file. The following example retrieves first some Analysis data from yesterday. Then, a second retrieval will extract the same data for 40 days ago run and append it at the end of the target file.
retrieve, class = od, type = analysis, stream = oper, expver = 1, date = -1, time = 12, levtype= pressure levels, levelis= 1000/850/500, param = temperature, target = "analysis.grb" retrieve, date = -40, target = "analysis.grb" |
Manipulate already retrieved data
Read requests can be used to filter or manipulate data from already retrieved files. It reads the data from a Unix file specified by the keyword source. This data can either be stored in a fieldset for further manipulation or written to a Unix file specified with the keyword target. Consider for example that file myfile contains certain meteorological parameters at all pressure levels. One can extract level 1000 hPa only and interpolate those fields to a 2.5x2.5 lat/lon grid, using the following MARS request:
read, source = myfile, levelist = 1000, grid = 2.5/2.5, target = only_1000 |
The verb retrieve can also be used to extract data from a Unix source file, perform post-processing on the result and store it into a target Unix file. In this case, retrieve needs all necessary directives as if it were a retrieval from the archives. Here is an example, assuming we want to extract geopotential and temperature Analyses for the 1st of January at 12 UTC:
retrieve, source = myfile, class = od, type = an, date = 20010101, time = 12, param = z/t, levtype = pl, levelist = 1000, grid = 2.5/2.5, target = only_1000 |
The advantage of retrieve is that it will fail if the requested data is not present in the source file. A disadvantage is that it has to specify all the directives, and the user might not know what are the exact contents of the source file.date = -1,